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Foreword 


The publication of the Standards and Guidelines for Quality Assurance in the European 
Higher Education Area (ESG), in May 2005, presented both an opportunity and a 
challenge for all organisations working in the field of quality assurance right across 
Europe. 

They are an opportunity, as for the first time, organisations are able to benchmark 
themselves against clearly defined cross-European standards. They also offer agencies 
the challenge of working with their European neighbours to interpret and implement 
the standards in the light of their own national and regional contexts. 

The Nordic Quality Assurance Network in Higher Education (NOQA) took up this 
opportunity in its 2005-2006 project, of which this report is the final outcome. Through 
the sharing of expertise, and the comparison of practices across the Nordic agencies, 
the project sought to interpret and clarify the ESG within the Nordic regional context. 
As a consequence it provides encouragement and information for all quality assurance 
organisations as they come to terms with the ESG. 

ENQA is pleased to publish this report as part of a series of occasional papers that 
deal with developments in European quality assurance. 

Peter Williams 
President, 

ENQA 


Preface 


Nordic Quality Assurance Network in Higher Education (NOQA) is a forum for 
information dissemination, exchanging experiences and pursuing projects of mutual 
interest. The main objective is to create a joint understanding of different Nordic 
viewpoints on issues related to higher education quality assurance. 

The network has a long tradition of conducting an annual joint project. This report 
presents the results of the 2005-06 project. The project is aimed at interpreting and 
clarifying the European standards and guidelines for quality assurance agencies, as 
agreed by the responsible ministers under the Bologna process. The project is also 
aimed at sharing and comparing practices among the Nordic agencies, for mutual 
inspiration concerning how organisations, processes and procedures could be enhanced 
in connection with the new European standards. 

The course of the project has in itself contributed to the fulfilment of these aims 
within the agencies of NOQA. On behalf of the project group, it is my hope that this 
report will inspire an even broader audience engaged in the enhancement of external 
quality assurance within higher education and in the implementation of the Bologna 
process. 


Tue V1NTHER-J0RGENSEN 
Project Chairman 


1. Summary 

This report presents the results of the joint 2005-06 project of the Nordic Quality 
Assurance Network in Higher Education (NOQA). The project focused on the European 
standards and guidelines for quality assurance agencies, examining them in a Nordic 
perspective. 

Purpose and focus 

The project aimed at interpreting and clarifying the European standards and 
guidelines for quality assurance, as suggested hy the European Association for Quality 
Assurance in Higher Education (ENQA) and agreed hy the responsible ministers 
under the Bologna process in Bergen in 2005. This was realised by studying Nordic 
practices; thus, the project’s aim was also to share and compare practices among the 
Nordic agencies for mutual inspiration concerning how organisations, processes and 
procedures could be enhanced in connection with the new European standards. 

The standards and guidelines in part 3 of the ENQA report, concerning external 
quality assurance agencies, have played a central role in the project. The project has 
included discussions and reflections on the standards under the following headings: 
Official status; Activities; Resources; Mission statement; Independence; External 
quality assurance criteria and processes used by the agencies; and Accountability 
procedures. 

In the report the standards and guidelines are discussed at a general level, 
focusing on the different aspects related to the wording of the text. Eurthermore, the 
requirements contained in each standard and its attached guidelines are discussed 
in detail. Examples of Nordic practices are presented in connection of each of the 
standards and guidelines. 

Outcome 

The discussions and reflections in the course of the project have pointed to a number 
of general dilemmas and uncertainties in the current set of standards and guidelines. 
These can be categorised under six themes: 

NATIONAL TRADITIONS AND LEGISLATION VERSUS EUROPEAN 
STANDARDS AND GUIDELINES 

The European standards and guidelines have been designed to be applicable to all 
quality assurance agencies in Europe, irrespective of their structure, function and size, 
and the national system they operate in. The project has brought about the experience 
that a quality assurance agency, nevertheless, must be regarded in the context of its 
national higher education system, its role within the quality assurance system and even 
the national culture and traditions. 

CONSISTENCY OF ASSESSMENT 

The ENQA report recommends that any European agency should, at no greater than 
five-year intervals, conduct, or be submitted to, a cyclical external review of its 
processes and activities. An assessment of whether the agencies are in compliance with 
the European standards for external quality assurance agencies should be included. 


The report suggests that national agencies should normally be reviewed on a national 
basis, respecting the subsidiarity principle. Assuring a consistent use of the standards 
becomes a challenge. The project has experienced that more precise threshold values 
regarding the different standards are necessary if the European agencies are to be 
reviewed and assessed in a consistent manner. A clarification of the status of the 
guidelines could be a first step in this direction. At the same time the different national 
contexts and models need to be respected. 

The value of informal practises 

The ENQA report states that the standards and guidelines focus more on what should 
be done, rather than how they should be achieved. Although in the actual wording 
of the standards, written documents and formal arrangements are given precedence 
over informal practices and arrangements. The project discussions, however, showed 
that it is important to consider both formal arrangements and more informal, yet well- 
established, practices to gain a reliable picture of the actual situation of an agency. The 
legal documents and other formal arrangements are necessary, but insufficient factors 
concerning the operations of an agency. 

DEFINITION OF CENTRAL CONCEPTS 

The standards and guidelines contain a considerable number of concepts assumed to 
be commonly used and understood by European agencies. In reality, the terms can 
be interpreted in different ways, and might gain new meanings as they are translated 
from English into other languages. The work with the standards and guidelines in this 
project has identified a number of terms which may need further clarification to make 
the use of the standards and guidelines more clear. 

THE IMPACT OF NON-COMPLIANCE WITH SPECIFIC STANDARDS 

The ENQA report explicitly demands that agencies should comply with all standards 
if they are to be included in the desirable sections of the planned register of external 
quality assurance agencies operating in Europe. The discussions during the project 
concerning the Nordic agencies’ compliance with the standards have shown that, 
although their operations in general have a very high level of compliance with the 
intensions behind the standards, due to e.g. national legislation specific operations 
and circumstances of minor importance can make compliance with certain standards 
questionable. This should be taken into consideration, e.g. by a Register Committee 
assessing the inclusion of agencies into the planned European register of quality 
assurance agencies. 

THE DEMANDS TO REVIEWS OF AGENCIES 

The ENQA report assumes that review processes of agencies will primarily be organised 
at the national level. An assessment of the credibility of the review process, and of the 
quality of the documentation in the review report, will become an important task for 
a Register Committee. 

The project demonstrated that written accounts need to be discussed and clarified in 
order to understand the actual nature of processes and procedures. The requirements 
of the standards and guidelines are easily interpreted from a national perspective, and 
the same terminology may not be comparable. This project suggests that the reviews 


of agencies should be thorough, and that it should be considered making a site-visit a 
mandatory element in the process in order to ensure the necessary documentation. 

Contribution 

The NOQA project 2005-06 raises more questions than it answers. Questions which in 
the view of the Nordic agencies should be dealt with at a European level as part of the 
implementation of the Standards and Guidelines for Quality Assurance in the European 
Higher Education Area. By examining the standards and guidelines in connection 
to Nordic practices, the project hopes to have contributed some insight into this 
discussion. The exercise has identified a number of different aspects that need to be 
considered in the process ahead. 


2. Introduction 


2.1 Background and purpose 

At the annual meeting in Copenhagen, May 2005, it was decided that the 2005-06 
NOQA project should focus on the Nordic agencies’ practices related to three themes: 
independence, follow-up procedures, and internal quality work. 

The project should aim to apply Nordic viewpoints to the development of these 
three issues, which are important to the credibility and effectiveness of external quality 
assurance processes. By making different national solutions known, and hy sharing 
experiences, it was an expressed hope that the Nordic agencies could inspire each other 
- and mayhe also other agencies around the world - with ways of coping with present 
and future challenges related to the themes. 

It was also decided that the project should incorporate the newly agreed European 
Standards and Guidelines for Quality Assurance in the European Higher Education 
Area in connection with the three themes. The European Association for Quality 
Assurance in Higher Education (ENQA) had proposed the standards and guidelines in a 
report', prior to the biannual meeting in Bergen 2005 for the ministers responsible for 
higher education in the Bologna signatory states. The Nordic project aimed to generate 
practical experience in evaluating current practices using the European standards and 
guidelines. 

It was an expressed desire that the examination of the themes, and the work with 
the related standards, would lead to a clarification of how the European standards 
and guidelines for external quality assurance agencies could be interpreted and made 
operational for assessment. In that sense, the project also aimed to contribute to 
discussions at a European level - e.g. the discussions about implementation of the 
planned register of European quality assurance agencies - and to internal discussions 
in other countries, e.g. countries preparing an application for inclusion of their national 
agency in the planned register. 

The purposes of the 2005-06 NOQA project can thus be summarised as follows: 

• interpreting and clarifying the European standards and guidelines for quality 
assurance agencies; 

• sharing and comparing practices among the Nordic agencies; 

• providing inspiration for quality assurance agencies in the light of European 
standards. 

2.2 Process 

The project has been divided into a number of stages. The first stage resulted 
in the production of national accounts describing the national practices related to 
independence, follow-up procedures and quality work in Denmark, Finland, Sweden 
and Norway. Prior to the drafting of the accounts, the project group had interpreted 
the relevant standards and guidelines into a number of criteria, guiding the work with 
the accounts. The accounts were written with reference to a common guide, requesting 
each agency to describe and assess current practices. 


^ European Association for Quality Assurance in Higher Education, Standards and Guidelines for Quality Assurance in the 
European Higher Education Area, Helsinki, 2005. 


In January 2006, the national accounts were discussed at a seminar in Stockholm. 
Twelve representatives from the agencies participated in lively discussions and 
reflections about national practices and their compliance with the standards. 

Already at this stage, it became clear that only to deal with some of the European 
standards and guidelines concerning quality assurance agencies in part 3 of the ENQA 
report was not an ideal approach. It was, however, only during the work on the first 
draft of this report that it became absolutely clear that all the standards and guidelines 
for agencies should be incorporated into the project, due to their strong interrelation. 

This implied an expansion of the project, with the associated gathering of new 
documentation, though not all standards and guidelines have been treated in the same 
depth. The national accounts and the information generated through the seminar in 
Stockholm still make up the core of the descriptions and discussions presented in this 
report. 

The findings presented in this report are based on documentation gathered 
throughout the course of the 2005-06 NOQA project, as well as discussions and 
reflections by the participating agency staff. The report does not express official 
viewpoints on behalf of the Nordic agencies, and the project group would like to stress 
that the report represents a first - and in no way final - step in the interpretation of the 
European standards and guidelines. 

2.3 Organisation of the project 

The project group consisted of one member from each of the five Nordic countries, plus 
an extra representative from Denmark: 

• Tue Vinther-Jorgensen, The Danish Evaluation Institute (EVA, Project Chairman) 

• Signe Ploug Hansen, (EVA) 

• Pirjo-Liisa Omar, The Finnish Higher Education Evaluation Council (FINHEEC) 

• Britta Lovgren, The Swedish National Agency for Higher Education (HsV) 

• Pal Bakken, The Norwegian Agency for Quality Assurance in Education 
(NOKUT) 

Asger ur Kjartansdottir, of the Ministry of Education, Science and Culture, Iceland, 
was originally a member of the group as well, but after having commenced a new 
position prior to the seminar in Stockholm in January 2006, Iceland decided to 
withdraw from the project. 

In addition to the members of the project group, the following persons participated 
in the seminar in Stockholm: 

• Anette Dorge Jessen (EVA) 

• Helka Kekalainen (FINHEEC) 

• Ossi Tuomi (FINHEEC) 

• Ragnhild Nitzler (HsV) 

• Staffan Wahlen (HsV) 

• Eva Liljegren (NOKUT) 

• Turid Hegerstrom (NOKUT). 

Staffan Wahlen also participated in the final meeting of the project group, as did Einar 
Hreinsson from the Ministry of Education, Science and Culture, Iceland. The group 
held four meetings between September 2005 and May 2006. 


2.4 Structure of the report 

The report consists of the foreword, a brief summary in chapter i, this introduction 
in chapter 2, and chapter 3 that describes some general lessons learned from working 
with the European standards and guidelines. The remainder of the report basically 
follows the order and structure of part 3 of ENQA’s report on standards and guidelines 
concerning the requirements for quality assurance agencies. 

Each of the standards is discussed in its own chapter. Only Standard 3.1 has not 
been included in the report, as this refers to the large number of standards concerning 
the quality assurance processes. Consequently, Standards 3.2 to 3.8 are discussed in 
chapters 4 toio, under the following headings, respectively: 

• Official status 

• Activities 

• Resources 

• Mission statement 

• Independence 

• External quality assurance criteria and processes used by the agencies 

• Accountability procedures. 

Each of these chapters begins by citing the text of the relevant standard and the 
attached guidelines. The standard and guidelines are then discussed at a general level, 
focusing on the different aspects related to the wording of the text. Afterwards, the 
requirements contained in the standard and its attached guidelines are discussed in 
detail. 

The project group has broken down each standard and its guidelines into a number 
of requirements. This has been done to clarify them for operational reasons. These 
requirements are presented as an introduction to the detailed discussions in each 
chapter. The breaking down of the standards and guidelines should not be viewed as an 
attempt to make a new order of priority, or to add new demands, but only as an attempt 
to make the text suitable for operational discussions - and potential assessment. One 
important feature, though, is that requirements originating from the standards and 
those originating from the guidelines are treated equally. This is due to the fact that 
the guidelines do, to a large extent, contain new requirements, and are not merely 
interpretations of the standards. 

The subsections of each chapter present the relevant practices of the four 
Nordic agencies, focusing on homogeneity and diversity among the countries. These 
presentations are only brief descriptions and do not aim to give a complete account of 
the practices and their context. The project group has, therefore, chosen to highlight a 
total of 15 national practices which have proven to be of interest to the other agencies 
in the course of the project. These examples are described in more detail in text boxes, 
in the hope that they might inspire development in the Nordic countries as well as in 
other countries. While some of the agencies might have similar practices, the project 
group has chosen to present examples for inspiration with a single agency as point of 
reference. 

When the four agencies are mentioned in the text, they are presented in alphabetic 
order - EVA, FINHEEC, HsV, and NOKUT. The same order is used when the countries 
are mentioned. The four agencies are referred to as “the Nordic agencies”, using the 
definite article, although Iceland has not taken part in the project. 


The project report was drafted with contributions from all members of the project 
group and edited by the two Danish members. The report was presented and discussed 
at the annual meeting of NOQA in Reykjavik, May 2006. 


3- Lessons learned 

The Standards and Guidelines for Quality Assurance in the European Higher Education 
Area have played a central role in the work of the 2005-06 NOQA project. This is 
especially the case for the standards and guidelines in part 3, concerning external 
quality assurance agencies. The work and discussions in the course of the project have 
pointed to a number of more general dilemmas and uncertainties in the current set of 
standards and guidelines. These dilemmas and uncertainties have been gathered and 
formulated into six questions below. The project group has not considered it its task to 
answer these questions, but does point to possible solutions where these have become 
apparent during the discussions. 

1. Do national traditions and legislation allow an agency not to comply with the 
European standards and guidelines? 

It is stated in the ENQA report presenting the standards and guidelines that they are 
designed to be applicable to all quality assurance agencies in Europe, irrespective of 
their structure, function and size, and the national system in which they operate. 

The NOQA project indicates that these aspirations can only be met, if the standards 
and guidelines for quality assurance agencies are perceived as addressing the whole 
national system of higher education, and not only the agencies as such. The project has 
brought about the experience that a quality assurance agency must be regarded in the 
context of its national higher education system, its role within the quality assurance 
system and even the national culture and traditions. Eor instance, it gives only little 
meaning to ask an agency to comply with the standards and guidelines if its national 
legislation distributes roles in the quality assurance system in such a way that the 
agency cannot operate in line with the European requirements. 

Thus, the standards and guidelines for quality assurance agencies do not only imply 
a challenge to agencies, but might also challenge institutions, governments and other 
stakeholders as well. In a review process, it seems to be an open question as to how 
to assess an agency operating under legislation which is not in line with European 
standards. Will it be necessary to change the national legislation, and later on the 
agency’s operations, before the agency can be said to comply with the standards 
and guidelines? Or should the national context be viewed as a reason for allowing 
exemption from the European standards when considering the compliance of the 
agency? 

2. How can a consistent assessment of the many European agencies' 
compliance with the standards and guidelines be assured? 

The ENQA report recommends that any European agency should, at no greater than 
five-year intervals, conduct, or be submitted to, a cyclical external review of its 
processes and activities. The reviews of agencies should include an assessment of 
whether the agencies are in compliance with the European standards for external 
quality assurance agencies. The report suggests that national agencies should normally 
be reviewed on a national basis, respecting the subsidiarity principle. Assuring a 
consistent use of the standards thus becomes a challenge. 


This report attempts to pin down the dimensions that should be taken into account 
when assessing an agency’s compliance with the many requirements. The project has 
experienced that more precise threshold values regarding the different standards are 
required if the European agencies are to be reviewed and assessed in a consistent 
manner. One very important issue to clarify in this respect is the status of the 
guidelines. In some cases, the guidelines can be viewed as attempts to establish 
threshold values that provide more detailed information on how the standard should 
be interpreted. In other cases, they are formulated as new standards without direct 
reference to the wording of the standard. 

In contrast, another experience of this project has been the importance of respecting 
the different national contexts and models when assessing an agency’s compliance with 
the standard. Therefore, it is equally important that the wording of the standards is 
generic and open to different systems, approaches, cultures and traditions. 

The discussion in the course of this project suggests that more guidance could be 
given to those working with assessing the European agencies’ compliance with the 
standards and guidelines, particularly a clarification of the status of the guidelines, but 
that this should be done in a balanced way respecting differences in national contexts 
and models. 

3. What is the value of informal practices when considering an agency's 
compliance with the standards and guidelines? 

The ENQA report states that the generic principle applied in the formulation of 
standards and guidelines has the consequence that these focus more on what should 
be done, rather than how they should be achieved. This is only followed to some 
extent in the actual wording of the standards, where priority is often given to written 
documents and formal arrangements, taking precedence over informal practices and 
arrangements. 

The discussions during the project have shown that it is important to take into 
account both formal arrangements and the more informal, yet well-established, 
practices in order to gain a reliable picture of the actual situation of an agency. The 
argument goes both ways. The legislation and other formal arrangements surrounding 
an agency can be in full compliance with the relevant standards, but this does not 
guarantee that the actual practice also is in line with the European requirements. For 
instance, an agency can be formally independent from ministries and other formal 
stakeholders, but not independent in practice if the government places a high level of 
pressure on the agency through informal channels. The opposite situation can also be 
found, where an agency with poor formal foundations is actually permitted to act with 
a very high level of autonomy and independence. 

The fact that legal documents and other formal arrangements are necessary, 
but insufficient factors concerning the operations of an agency, is a challenge that 
must be dealt with in the procedures and methods applied in the reviews of the 
European agencies. There is no doubt that the assessment process will become more 
complicated - and the determination of threshold values more delicate - when informal 
arrangements and practices are to be taken into consideration. 


4. How can the terminology used in the standards and guidelines become 
clearer? 

The standards and guidelines contain a considerable number of words assumed to be 
commonly used and understood by European agencies. In reality, the terms can be 
interpreted in different ways, and might very well gain different meanings as they are 
translated from English into other European languages. 

The work with the standards and guidelines in this project has pointed to a number 
of terms which may need further clarification. One example is the term policy. What is 
a policy, and what should be the minimum requirements for the content of a document 
if it is to qualify for the label policy? Another example is the term procedures. Can 
procedures be well-established habits or cultures, or must the way to conduct operations 
be described in writing if it is to be accepted as a procedure? If the latter is the case, 
yet another question is how long must it have been in operation to be accepted as more 
than just a piece of paper? A similar line of questioning arises concerning the terms: 
goals and objectives, legal basis, management plan, mechanisms, etc. 

Reflections on the meaning of these terms might be useful to both agencies and their 
assessors, but it is equally important that such reflections do not result in formulations 
that are too narrow and prescriptive. 

5. Must an agency comply with all standards in order to be considered as 
being in compliance with the European standards and guidelines? 

The ENQA report explicitly demands that agencies should comply with all standards 
if they are to be included in the desirable sections of the planned register of external 
quality assurance agencies operating in Europe. This might imply that very few agencies 
could be accepted as being in compliance with European standards. 

The discussions during the project concerning the Nordic agencies’ compliance 
with the standards have shown that, although their operations in general have a very 
high level of compliance with the intensions behind the standards, specific operations 
and circumstances of minor importance can make compliance with certain standards 
questionable. This can, for instance, be due to national legislation, e.g. legislation 
concerning the role of an agency in the follow-up on external quality assurance 
processes. 

The experiences of this project suggest that while a review process conducted by 
a panel of peers or experts should include an assessment of the compliance with the 
individual standards and their attached guidelines, it should be possible for a Register 
Committee to make an overall assessment, distinguishing between vital and less vital 
requirements of the European standards. 

6. Which demands should be made to assure a credible review process, 
including assessment of the agencies' compliance with the European 
standards and guidelines for external quality assurance agencies? 

The ENQA report assumes that review processes of agencies will primarily be organised 
at the national level. Although a growing interest for reviews organised by ENQA is 
notable, an assessment of the credibility of the review process, and of the quality of 
the documentation in the review report, becomes an important and separate task for 
a Register Committee. 


An important experience gained from producing national descriptions and 
assessments in this project is that such written accounts need to be discussed and 
clarified in order to understand the actual nature of processes and procedures, etc. Each 
agency tends to interpret the requirements of the standards and guidelines from their 
own national perspective, and the same terminology might have different implications 
and meanings in different countries^. 

This project suggests that the reviews of agencies should be thorough, and that it 
should be considered making a site-visit a mandatory element in the process in order 
to ensure the necessary documentation. This would enable the peer review group, or 
expert panel, to gain a deeper understanding of the working mode of the agency subject 
to review, including its interpretations of the standards and guidelines. 

The independence of the peers or experts conducting a nationally organised review 
of an agency will, of course, be another important issue for a Register Committee to 
consider. 


^ This experience is very much in line with the conclusions in Crozier, Fiona et al., Quality Convergence Study, ENQA Occasional 
Papers 7, Helsinki, 2005. 


4- Official status 


European standard 3.2: 

Agencies should be formally recognised by competent public authorities in 
the European Higher Education Area as agencies with responsibilities for 
external quality assurance and should have an established legal basis. They 
should comply with any requirements of the legislative jurisdictions within 
which they operate. 

There are no guidelines attached to this standard. 


4.1 About official status 

External quality assurance is seen as a major tool for creating confidence and trust 
in academic standards and frameworks of qualifications, as the Bologna process 
progresses towards a common European Higher Education Area in 2010. Therefore, 
an important issue is what kind of organisations should be entrusted with the task of 
conducting external quality assurance processes. 

Standard 3.2 states that these organisations should be agencies with official status, 
and that these agencies should fulfil some more detailed requirements in order to be 
viewed as organisations with an official status. In Denmark, Einland, Sweden and 
Norway, external quality assurance of higher education has almost exclusively been a 
task for the public agencies EVA, EINHEEC, HsV, and NOKUT respectively (and their 
predecessors if any), as an integrated part of the national quality assurance system. The 
question of official status has, therefore, been of less interest until recently. 

With the increased need for transparency across national boarders in a more open 
European landscape, it is natural that minimum standards are formulated in order to 
ensure reliable processes in countries or regions which have not yet established a system 
for external quality assurance. 

At the same time, the emphasis on the public recognition and legal bases of the 
agencies suggests that external quality assurance in the future could be performed 
by organisations which are not, by definition, part of the national quality assurance 
system of higher education. The call for compliance with the legislative requirements of 
the jurisdiction in which they operate implicitly paints a picture of organisations with 
activities across national boarders. The standard anticipates a situation where higher 
education institutions can choose more freely which agency they want to cooperate 
with. 

A European Parliament and Council Recommendation from Eebruary 2006 falls 
in line with this approach to external quality assurance of higher education. The 
Parliament and the Council recommend that EU member states enable higher 
education institutions to choose an agency which meets their needs and profile among 
quality assurance agencies included in the European Register, provided that this is 
compatible with their national legislation or is permitted by their national authorities 
(2006/143/EC). 



The Danish university act of 2003, in principle, already makes it possible for 
universities to choose other organisations than EVA to carry out the external 
quality assurance of their programmes, provided that these organisations comply with 
international criteria for quality assurance agencies. 

If this model becomes widespread, it is natural that the European standards and 
guidelines limit that choice to organisations that are recognised by competent public 
authorities and have an established legal basis in at least one of the Bologna member 
states. 

The requirements of the European standard regarding official status will be 
discussed under the following three headings in the next sections: 

• Recognition and established legal basis 

• Compliance with requirements of jurisdiction 

4.2 Recognition and established legal basis 

The agency should be formally recognised by competent public authorities in the European 
Higher Education Area as an agency with responsibilities for external quality assurance 
and should have an established legal basis. 

The above formulation does not offer any explanation as to who competent public 
authorities may be. An obvious interpretation is that the relevant public authorities 
in the country (or legislative jurisdiction) within which an agency operates, must 
authorise it to do so. In that case, an agency operating across borders in the 
European Higher Education Area would need to ask the relevant Ministry or another 
national authority for recognition as an agency with responsibilities for external quality 
assurance of higher education in the specific country. 

Another possible interpretation is that the standard, with its wording, refers to 
the planned European register of quality assurance agencies. If that is the case, only 
agencies accepted for the register can operate external quality assurance processes in 
the signatory states of the Bologna process. This would also imply that once included 
in the register, an agency should have the possibility to carry out its processes within 
the whole European Area. 

It is, of course, important to clarify the legal basis of an agency when assessing its 
official status. Even though they are all government bodies, the Nordic agencies have 
different kinds of legal foundations: 

• EVA was established in law by the Danish parliament in 1999 as an independent 
institution under the auspices of the Danish Ministry of Education, with its 
predecessor having already operated since 1992. Three legal documents regulate 
EVA’s activities: The Danish Evaluation Institute Act and two ministerial 
regulations. 

• EINHEEC was established in 1995 and is governed by a decree. EINHEEC operates 
under the Einnish Ministry of Education. 

• HsV is a state agency established in 1995 by the Swedish government (a 
predecessor already operated in 1992). Its operations are determined by different 
official documents, e.g. the annual budget directive and the ordinance, which 
contains instructions for the agency. 


• NOKUT was established in 2003 by the Norwegian parliament as part of a reform 
of Norwegian higher education, and its activities are regulated by chapter 2 of the 
University and College Act, and in ministerial regulations. 

All the Nordic agencies are established by law or decree as part of the national quality 
assurance system for higher education. Therefore, the question about public recognition 
is somewhat speculative at this stage, and will not be dealt with any further here. 

4.3 Compliance with requirements of jurisdiction 

The agency should comply with any requirements of the legislative jurisdictions within 
which it operates. 

None of the Nordic agencies have yet gained international experience as a formal 
quality assurance agency. Naturally, the Nordic agencies all comply with the 
requirements in their homecountries. Therefore, the question about compliance with 
the requirements of other legal systems is also somewhat speculative at this stage. 


5- Activities 


European standard 3.3: 

Agencies should undertake external quality assurance activities 
(at institutional or programme level) on a regular basis. 

Guidelines: 

These may involve evaluation, review, audit, assessment, accreditation or other 
similar activities and should be part of the core functions of the agency. 


5.1 About activities 

The expectations expressed in standard 3.3, and the guidelines to the standard, signal 
that any organisation undertaking quality assurance activities cannot automatically be 
considered a quality assurance agency. To qualify for being considered as such, the 
organisation must be able to demonstrate a degree of regularity in its quality assurance 
activities, and that such activities are core functions of the organisation. In that way, 
the standard makes clear that credible management of external quality assurance 
activities requires a specific kind of professionalism and experience in the field. 

The guidelines specify the meaning of the term “activities” by stressing that 
activities may involve evaluation, review, audit, assessment, accreditation or other 
similar activities. In that sense, it seems relatively straight forward to assess whether an 
agency complies with the standard, as a description of the activities of an agency would 
provide relevant and sufficient documentation. 

As the remaining sections of this chapter will reveal, the terms “regular basis” and 
“core function” do, however, create some challenges to an assessment process. Section 

5.2 discusses relevant interpretations of the former term, and section 5.3 deals with 
the latter. 

5.2 Activities on a regular basis 

Agencies should undertake external quality assurance activities (at institutional or 
programme level) on a regular basis. 

There is a broad similarity in the mix of quality assurance activities undertaken by 
the Nordic quality assurance agencies. All four agencies are presently conducting both 
accreditations and evaluations. Audits are, at present, conducted by EVA, FINHEEC 
and NOKUT and have, until recently, also been part of the activities of HsV, and will be 
included in the next cycle. The objects and focuses of one or more of the activities are 
also quite similar among the Nordic agencies, in the sense that all four agencies have 
programmes as the object of one or more of their quality assurance activities. Thematic 
evaluations are also a common activity among all four agencies. Focus on institutions 
as objects of the quality assurance activities exists among all the agencies presently 
engaged in audits, and in thematic reviews as presently carried out by HsV. 



The Nordic agencies undertake quality assurance activities on a regular basis, and 
these activities are planned and carried out continuously at all four agencies. This has 
been the case in all four countries for a considerable number of years. 

5.3 Activities as a core function 

External quality assurance activities, e.g. evaluation, review, audit, assessment, 
accreditation or other similar activities, should be part of the core functions of the agency. 

The expectation that the quality assurance activities are part of the core functions 
of the agency may be interpreted in different ways. A central question is whether 
this guideline is to be interpreted quantitatively, in the sense that fulfilment requires 
that most of the financial and human resources are spent on the quality assurance 
activities of the agency, or whether a more qualitative approach - or a combination of 
a quantitative and qualitative approach - would be more appropriate. Whichever the 
choice, a threshold for the level of acceptance has to be established. The documentation 
required for the assessment of compliance will naturally depend on the choice of 
approach (quantitative or qualitative). 

Quality assurance activities are a core function of all the Nordic agencies. Whether 
the quality assurance activities represent the core function of the individual Nordic 
agencies, or form part of the core functions of the agencies, reflects the mandate of 
the different agencies and thus the extent to which they undertake activities other than 
quality assurance, defined as evaluations, audits, accreditations, and similar activities. 

The activities of EVA and FINHEEC are all related to external quality assurance, 
whereas both HsV and NOKUT also have an important function as centres for the 
recognition of foreign degrees. HsV has further tasks concerning statistics and analysis, 
planning and research, and information and legal supervision within higher education 
in Sweden. 

Whereas the legal frameworks of EVA, EINHEEC and NOKUT implicitly or explicitly 
emphasise quality assurance activities as the core function of the agencies, the legal 
framework of HsV involves several tasks, but, in terms of activities, staff and resources, 
quality assurance is the core function. HsV has experienced that the quality assurance 
activities benefit from the agency’s other activities, e.g. student statistics. 

The tasks and functions of the Nordic agencies illustrate the importance of taking 
into account the mandate and context of an agency when considering the extent to 
which it complies with the requirement. Quality assurance activities should be part of 
the core functions of the agencies, but an agency can very well have other important 
tasks and functions which might even complement the quality assurance activities. 
What constitutes a core function must, therefore, be a question to be answered in 
each specific situation by considering a mixture of elements, e.g. the applied resources, 
percentage of employees, the status in the organisation, complementarity with other 
functions, etc. 


6. Resources 


European standard 3.4: 

Agencies should have adequate and proportional resources, both human and 
financial, to enable them to organise and run their external quality assurance 
process(es) in an effective and efficient manner, with appropriate provision 
for the development of their processes and procedures. 

There are no guidelines attached to this standard. 


6.1 About resources 

The credibility of an agency is very much dependent on its ability to perform its role 
in the national quality assurance system in an effective and efficient manner. Without 
the necessary resources, an agency cannot contribute to the assurance of quality 
in higher education programmes and institutions. Therefore, access to adequate and 
proportional resources - both human and financial - is highlighted as a requirement in 
the European standards and guidelines. 

The words adequate and proportional indicate a threshold for the minimum 
resources allocated to an agency, but also imply that their application takes into account 
the national context and the nature of the quality assurance processes conducted 
by the agency. The requirement for agencies to have appropriate provisions for the 
development of their processes and procedures can also be viewed as a point of focus 
when assessing the resources of an agency. 

Standard 3.4 will be discussed in the following two sections, which focus on 
financial and human resources respectively. 

6.2 Financial resources 

The agency should have adequate and proportional financial resources to enable it 
to organise and run its external quality assurance process(es) in an effective and 
efficient manner, with appropriate provision for the development of its processes and 
procedures. 

The agencies in Denmark, Finland, Norway and Sweden are primarily funded by 
government grants. 

• The scale of EVA’s government grant is determined each year through the general 
finance act. The government grant makes up 92% of the revenue of the agency. 

The remaining 8% comes from commissioned work funded by external sources, 
e.g. ministries and institutions. All evaluations and other activities initiated by 
EVA are financed through EVA’s government grant. The management draws up 
EVA’s budget, which must be approved by the board and the minister of education. 
The government grant consists of two entries: salaries and general operating 
expenses. EVA can transfer from salaries to operating expenses, but not vice versa. 

• The Ministry of Education allocates FINHEEC’s operating funds one year at 

a time, following negotiations. In principle, the council decides independently 
its plan of action, but the ministry may make the final decision on the 



implementation through its funding decision. In practice the tasks of FINHEEC 
are adjusted to match the resources. 

• HSV receives a total grant each year for all its activities. The University Chancellor 
decides on the allocation of funds to the various aspects of the agency’s operations. 
The Department of Evaluation has the largest budget of all the departments for 

its activities. 

• NOKUT obtains its funding from the Ministry of Education. NOKUT decides how 
the funding is allocated. NOKUT has experienced an increase in funding over 
recent years. 

As the Nordic agencies are all state funded, the question of financial resources is 
thus linked to the question of potential government interference in operations. The 
independence of an agency can indirectly be limited if its financial resources are scarce, 
or if they are accompanied by restrictions on how they may be used. Eor instance, 
a broad and strong mandate to evaluate higher education institutions can be of only 
limited value if the agency cannot afford the costs of hiring experts, paying site-visit 
expenses or perhaps conducting surveys, etc. Another consideration is that ministries 
(or higher education institutions) may gain some influence if they, through extra grants, 
can select programmes to be subjected to external quality assurance. Einally, grants can 
be earmarked for specific purposes, which in some cases can reduce the effectiveness 
and efficiency, as well as limit the operational independence of an agency. 

The Nordic agencies receive annual grants at quite different levels, but it is difficult 

- if not impossible - to use the actual amounts as indicators for whether the resources 
are adequate and proportional. This is primarily due to the fact that the mandate and 
the nature of quality assurance processes differ considerably between the agencies. 

Eor instance, the grant allocated to HSV by the Swedish government not only covers 
expenses related to external quality assurance of higher education, but also finances 
the other mandatory tasks of the agency, e.g. recognition of foreign degrees, student 
information and analysis services for the government and parliament. In the Danish 
case, EVA’s government grant not only finances quality assurance activities within the 
higher education sector, but in the school sector as well. 

Even if it was possible to isolate funds allocated to quality assurance activities within 
the higher education sector, this would not provide a reliable basis for the assessment 
of financial resources. The adequacy and proportionality of the resources has to be seen 
in conjunction with the demands made by the chosen methodologies, processes and 
procedures. Eor instance, NOKUT can carry out far more programme accreditations 
than institutional audits for a given amount of money. And the Einnish approach to 
institutional audit might require more resources than the Norwegian one - or vice versa 

- which makes direct comparisons a dangerous exercise. 

The grant can also be allocated with certain restrictions on how it is to be used, 
or as a lump sum as in the case of NOKUT and HSV. Again, this implies that the 
actual amount of the annual grant is insufficient documentation for the assessment 
of an agency. 

The only way forward seems to be a pragmatic one, with at least three steps. The 
first step is to assess whether the financial resources allows the agency to carry out 
its mandate effectively in terms of its current portfolio of quality assurance processes. 
Effectiveness in this respect must be seen as a question of the agency’s ability to 


reach a satisfactory share of the national higher education sector with its quality 
assurance processes, within an acceptable timeframe. The degree of systemisation and 
coverage are key words in the assessment of this. For instance, if an agency shall, 
by mandate, conduct institutional audits and programme accreditations, the assessors 
must estimate whether the agency is able to process all relevant institutions and all 
relevant programmes within a reasonable number of years. The number of audits and 
accreditations carried out over the previous years can serve as an indicator. 

The length of the timeframe could be another indicator. The European standards 
and guidelines call, in part 2, for external quality assurance to be undertaken on a 
cyclical basis, with a predefined length of cycle. The predefined length of cycle would 
then be the timeframe to use in the assessment of resources. The Finnish audits of 
universities and polytechnics are planned to be conducted at six-year intervals, but in 
practice FINHEEC can only audit all the Finnish higher education institutions on time 
if the Ministry of Education allocates sufficient, annual resources. Also EVA, HsV, and 
NOKUT use six-year intervals in the planning of cyclic processes. 

The second step is to assess whether the agency has sufficient resources to run 
each individual quality assurance process effectively. Effectiveness in this respect must 
primarily be viewed as a question of the ability of the agency to gather, analyse, 
and assess documentation in a way that imparts credibility to any conclusions and 
recommendations in its reports. The solidity of the documentation material and the 
transparency of the assessment process would be in focus here. The whole of section 2 
of the European standards and guidelines and standard 3.7 provide hints as to potential 
indicators and threshold values, e.g. that agencies should employ the submission of a 
self-evaluation report, the appointment of a group of experts, and the completion of a 
site visit as integral elements in their processes for external quality assurance of higher 
education. 

The third step is related to the requirement that agencies have appropriate provision 
for the development of their processes and procedures. The agency must have sufficient 
resources to adjust current processes and procedures, and develop new approaches to 
external quality assurance. In an assessment process, it would be relevant to investigate 
the capability to conduct internal quality assurance activities, as required by standard 
3.8, and to use the results for enhancement and development. It would also be relevant 
to focus on the process that leads to the introduction of new external quality assurance 
schemes. For instance, both FINHEEC and NOKUT conducted extensive consultation 
processes prior to completing and introducing new audit concepts in Finland and 
Norway respectively. EVA and HsV are currently conducting similar consultations. 

6.3 Human resources 

The agency should have adequate and proportional human resources to enable it 
to organise and run its external quality assurance process(es) in an effective and 
efficient manner, with appropriate provision for the development of its processes and 
procedures. 

Besides financial resources, standard 3.4 also requires agencies to have adequate and 
proportional human resources. This does to some extent overlap with the requirement 
for financial resources, as an agency can and does use financial grants to hire staff. 
Therefore, the emphasis on the need for human resources must imply a requirement 


that agencies have a solid organisation capable of managing and administrating quality 
assurance processes. 

The question of adequate and proportional human resources not only concerns the 
number of employees at an agency. The competences of the staff, and the way in which 
an agency ensures the continuous development of these, are equally important. 

The Nordic agencies have different recruitment strategies and, hence, also different 
human resource profiles: 

• EVA’s evaluation consultants all have master’s degrees or teaching degrees with 
supplementary education, but EVA does not currently have PhD’s among its staff. 
Students are employed to assist the consultants in certain processes. EVA hires 
persons with comprehensive work experience from within the education sector, 
relevant ministries, etc., but also hires graduates and other personnel based on 
their methodological and personal competences. 

• All staff members at EINHEEC, except for clerical staff, are in practise required to 
have Master’s degrees. The Secretary General of the Council is required to hold a 
PhD, and also other employees have PhD’s. Prior evaluation experience is required 
of Senior Advisers. Most staff members have prior work experience from either 
higher education institutions or education administration, or both. In recruiting 
project staff, subject knowledge and evaluation experience are recommendable. 
Additional experience in evaluation is gained through an apprenticeship-type 
introductory phase. It is the Secretariat’s policy to hire staff with different 
educational backgrounds. 

• The Evaluation Department at HsV is the largest department of the agency. About 
40 per cent of its staff today have licentiate degrees or PhD’s. As HsV has a number 
of other responsibilities in addition to evaluation, the competences of staff from 
other departments can also be used in the evaluation activities. The strategy of 
the agency is to recruit people with different academic backgrounds and ages, and 
to maintain a balanced gender distribution. All staff must also have experience 

of the university sector, preferably also of evaluation. They must possess “social 
competence” and be proficient writers. 

• Most NOKUT employees are educated to at least Master’s degree level. New 
staff have competences based on at least one of the following aspects: evaluation 
experience, knowledge and experience of the higher education sector. 

The different staff profiles of the Nordic agencies, ranging from students to PhD’s, 
indicate that tasks and needs can be viewed differently, although processes and 
procedures are, to a large extent, uniform. One reason for this is different national 
traditions, e.g. for the division of labour between agency staff, experts and institutions 
in the quality assurance processes; how legitimacy is guaranteed in the processes; the 
length and types of reports; etc. Thus, different staff profiles are not necessarily simply 
a result of sufficient or insufficient human resources. Any assessment process must seek 
to clarify the reflections an agency has made on the nature of the actual tasks and the 
related needs in terms of staff qualifications and competences. It would, of course, be 
preferable if the agency could document such reflections. 

An assessment of human resources must examine whether the agency has 
mechanisms in place to ensure that staff competences conform to current tasks and 
needs. Are the reflections clear and explicit, and are they applied to a systematic 


development of the staff competences? Hence, to comply with the standard, an agency 
should be able to present documentation that competence development of its staff is 
both a priority and systematic. Relevant sources of documentation in the process of 
assessing an agency’s fulfilment of these requirements are policies and procedures for 
staff recruitment, policies or concepts for staff development, accounts for competence 
development activities over a number of years, etc. A clear danger to competence 
building is high staff turnover in an organisation. Therefore, the degree of staff 
turnover in an agency should also be examined. 

All the Nordic agencies have mechanisms in place to ensure systematic staff 
development: 

• At EVA, development of staff competences is highly prioritised, and a concept 
for systematic staff development has been in place for years. The individual 
employee’s competence profile and need for future development are discussed in 
a personal development interview once a year. The purpose of this interview is to 
ensure the continuous development of the competences of both the employee and 
the organisation as a whole. The development of competences is partly based on 
the individual employee’s current needs and wishes, and partly on EVA’s goals and 
needs. Relevant areas for development are identified through a competence profile 
clarification, based on a competence chart. 

• At EINHEEC, staff competencies and development are discussed and planned 
according to current tasks in annual result and development discussions between 
the Secretary General and the staff members. An internal document specifies 
the responsibilities and assignments for each staff member. The document is 
revised annually following the result- and development discussions. Staff policy 
is to encourage training and staff development. An inherent risk in competence 
development and the task of creating a learning organisation is high staff turnover 
and the use of fixed-term contracts (max. 4 years). 

• HsV’s policy of staff development defines staff development as a tool that includes 
everything that enhances the competence of the staff members, e.g. new and more 
advanced tasks; introduction programmes; courses; seminars; and participation 
in network, supervision or mentor programmes. The policy states that each staff 
member is responsible for his/her own development, which should be planned in 
development discussions with the head of department. The procedures for staff 
development at HsV are described in more detail below - see text box. 

• NOKUT has several mechanisms to develop staff competences. NOKUT’s policy 
for staff development is described in a plan. The plan states that development 

of competences is the responsibility of each unit, and that each employee has 
a responsibility for his/her own development. Another important element is the 
mandatory annual meeting between each employee and his/her manager, where 
discussions about the development of competences must take place. The plan also 
states that the effects of the efforts shall be evaluated by the units, annually. 


Staff development at HsV 

The evaluation department has drawn up an inventory of the competence 
development needs of its staff and the possibilities open to them. This inventory 
is accessible via the evaluation department’s Intranet. The purpose of the 
inventory is to make staff development possibilities known to everybody and 
to achieve a shared understanding of the kinds of competences required to 
work in the evaluation department. 

The evaluation department has organised its operations on a project basis. 
The department has identified that the staff should have competences in four 
areas: general project management; evaluation theory and practice; pedagogy; 
and knowledge of national and international higher education, and related 
areas. 

The general project management competence could include the following 
areas: project management; budgets and financial management; computer 
skills; knowledge of the higher education sector; knowledge of public 
administration; interviewing techniques; oral presentation; and languages. 

Pedagogical competence could include group dynamics and learning. 
Knowledge of national and international higher education, and related areas, 
could include taking part in reference groups; taking part in conferences 
and networks; international surveys; and staff exchange with other evaluation 
organisations. 

Some of the staff development activities are arranged for the staff as a 
whole, some on an individual basis. Furthermore, several thematic seminars 
have been held in conjunction with the Department’s weekly meetings. Some 
of the themes addressed include: working in projects; the evaluation model; 
international issues. 


7- Mission statement 


European standard 3.5 

Agencies should have clear and explicit goals and objectives for their work, 
contained in a publicly available statement. 

Guidelines: 

These statements should describe the goals and objectives of agencies’ quality 
assurance processes, the division of labour with relevant stakeholders in higher 
education, especially the higher education institutions, and the cultural and 
historical context of their work. The statements should make clear that the 
external quality assurance process is a major activity of the agency and that 
there exists a systematic approach to achieving its goals and objectives. There 
should also be documentation to demonstrate how the statements are translated 
into a clear policy and management plan. 


7.1 About mission statement 

The requirement to have a publicly available mission statement aims at enhancing 
transparency and the level of information. Having official status with an established 
legal basis is no longer sufficient, as the national framework for higher education is 
gradually being complemented by a European one. A mission statement can also help to 
improve transparency and accountability at the national level as well. 

The guidelines provide more detailed requirements concerning what must be 
included in a mission statement. Apart from the goals and objectives of the quality 
assurance processes, the statement should describe the division of labour between 
relevant stakeholders, as well as the history and cultural context of the agency in 
its national quality assurance system. These descriptions can be of great assistance 
to foreign authorities, institutions or others who want to understand the nature of 
the organisation. In the national higher education community, this information will 
presumably be well known and might be of less interest, but it may nevertheless help 
to enhance understanding of the agency’s work if this often scattered information were 
assembled in a single document. 

Furthermore, the guidelines request agencies to adopt a systematic approach to 
achieving their goals and objectives and to demonstrate that the goals and objectives 
of the mission statement are translated into a clear policy and management plan. In 
other words, the management must work strategically and have converted the general 
goals and objectives of the agency to a more operational level to fulfil the requirements. 
The operational documents should, of course, be in line with the general goals and 
objectives, and the coherence between the two should be subject to examination when 
assessing compliance with the requirements. The operational documents can serve 
as the background for the accountability procedures of the agency, as required in 
European standard 3.8. 

Standard 3.5 and the attached guidelines can be divided into three requirements, 
focusing on: 



• Clear and explicit goals and objectives 

• Division of labour, context and activities 

• Systematic approach, clear policy and management plan 

7.2 Clear and explicit goals and objectives 

The agency should have clear and explicit goals and objectives for its work, especially its 
quality assurance processes, contained in a publicly available statement. 

Standard 3.5 does not demand that agencies have a document entitled Mission 
Statement, but, taken literally, it does ask for a single document with the necessary 
information. To be publicly available, one should expect to find the statement accessible 
on the agency’s website. If the statements are to enhance transparency across boarders 
in the European Area for Higher Educations, it would be an advantage if they were 
available in English as well as in the national language(s). 

The Nordic agencies are not yet familiar with the idea of having a mission statement. 
Many of the elements supposed to be contained in a mission statement will have to 
be transferred from other public documents. As public institutions, the general goals 
and objectives of the Nordic agencies, and their quality assurance processes, are to a 
large extent described in their legal bases, e.g. laws, decrees, etc. (see the descriptions 
under standard 3.2 regarding official status). More detailed descriptions of the specific 
objectives of cyclic processes might be described in the official documents authorizing 
the agencies to undertake these. 

None of the Nordic agencies have documents explicitly named Mission Statement, 
but they all have documents with goals and objectives for their work and quality 
assurance processes available on their websites: 

• EVA’s Strategy Plan 2004-06 was adopted by its board. It is available in Danish 
and English. 

• Eor each four-year term, the council sets an action plan, in which EINHEEC’s 
goals and objectives are specified for the term. EINHEEC’s Action Plan 2004-07 
is available in Einnish, Swedish, and English. 

• HsV has its own internal documents, such as an activity plan and a more visionary 
document - the platform - adopted by the University Chancellor, that lay down 
the perspectives and approaches that are to characterise both its evaluation 
activities and the Agency’s other operations. 

• NOKUT’s board has adopted a strategy plan with goals and objectives for its work. 
The plan has no fixed time schedule, but is reconsidered and revised - if necessary 
- once a year. The plan is available in Norwegian. 

This indicates that it is not the title and format, but the content of central policy 
and strategy documents that should be in focus when working with the standard. A 
central question is whether goals and objectives are presented to the public in one 
key document, or whether this information is scattered in a number of documents. 
Another important question is the status of the documents. Have they been processed 
and adopted by the board, council or chancellor - if any - and have they been 
communicated actively to the higher education society? 

Although the websites of the Nordic agencies all have supplementary explanations 
in English, it could ameliorate the communication to foreign stakeholders if goals and 


objectives as well as other crucial information were summarised and gathered in an 
easily accessible mission statement in English. 

7.3 Division of labour, context and activities 

The statement should describe the division of labour between relevant stakeholders 
in higher education, especially the higher education institutions, and the cultural and 
historical context of the agency’s work, and should make clear that the external quality 
assurance process is a major activity of the agency. 

The documents mentioned in 7.2 all contain general descriptions of the division of 
labour between higher education institutions and other stakeholders. The cultural and 
historical contexts of the work of the agencies are also described to various degrees. 

The readers of the documents are left with absolutely no doubt that external quality 
assurance is at the core of the agencies’ activities. 

A look across the Nordic documents shows that the requirements for more detailed 
information about the agencies can be fulfilled in many ways. The division of labour 
between the agency, higher education institutions and other stakeholders can be 
described explicitly in a separate section of the document, or take a more implicit form 
as part of a general description of the tasks and procedures of an agency. The context 
can be presented as part of an introduction, or as an analysis of developments in higher 
education and subsequent challenges to external quality assurance agencies, e.g. in 
relation to the Bologna process. And the focus on external quality assurance can make 
it very clear that such processes are a major activity of an agency although this is not 
explicitly stated in the document. 

When working with this requirement, one should not only take explicit headlines 
and statements into account. It is necessary to make a general assessment of whether 
the documents provide a satisfactory explanation of the background, context and role of 
the agency within the higher education system in which it operates. 

7.4 Systematic approach, clear policy and management plan 

The statement should make clear that there exists a systematic approach to achieving 
goals and objectives, and the agency should be able to demonstrate how the statement is 
translated into a clear policy and management plan. 

The requirements presuppose that an organisation has goals and objectives at a 
high level of abstraction, which are interpreted into more concrete policies and plans. 
The Nordic agencies use their own terminologies and understanding of the hierarchy 
between goals, objectives, policies and plans, but still with a systematic approach. 

For instance, both EVA and NOKUT have chosen an approach starting from a vision 
and basic values at a high level of abstraction, via goals and main strategies, to more 
concrete strategies and objectives for the various fields of action. 

The policies and management plans can be an integral part of the key documents 
mentioned in 7.2, or be separate documents. FINHEEC’s Action Plan - see text box 
- contains a plan for the activities in the term of the current council, whereas EVA 
and HsV have made management plans covering whole cycles of quality assurance 
activities as separate documents, supplemented by annual action plans. For example, 
after consultations with the higher education institutions, HsV established an action 
plan for subject and programme evaluation for the period 2001 - 2006, which was 


further detailed in each year’s action plan. A process for 2007 - 2012 is now under 
way. 

A critical issue when examining the practices of an agency is whether the initiative 
to commence quality assurance processes comes from the agency or from the 
institutions. In the latter case, the possibilities for making a management plan covering 
a whole cycle are reduced. For instance, the Norwegian institutions must apply for 
an accreditation by NOKUT, and, therefore, NOKUT has a more responsive planning 
concept depending on the annual number of applications. 


FINHEEC’s Action Plan 2004-07 

For each four-year term, FINHEEC sets an Action Plan, which details the 
council’s tasks, operational policy, objectives and principles of operation for 
the ongoing term. It also specifies how these are operationalised into practice 
by detailing the quality assurance activities and evaluations performed during 
the term. In addition, a preliminary schedule for evaluations undertaken during 
the term is included. The action plan is revised annually. 


8. Independence 


European standard 3.6: 

Agencies should be independent to the extent both that they have autonomous 
responsibility for their operations and that the conclusions and recommendations 
made in their reports cannot be influenced by third parties such as higher 
education institutions, ministries or other stakeholders. 

Guidelines: 

An agency will need to demonstrate its independence through measures, such as: 

• Its operational independence from higher education institutions and governments 
is guaranteed in official documentation (e.g. instruments of governance or 
legislative acts). 

• The definition and operation of its procedures and methods, the nomination and 
appointment of external experts and the determination of the outcomes of its 
quality assurance processes are undertaken autonomously and independently from 
governments, higher education institutions, and organs of political influence. 

• While relevant stakeholders in higher education, particularly students/learners, 
are consulted in the course of quality assurance processes, the final outcomes of 
the quality assurance processes remain the responsibility of the agency. 


8.1 About independence 

The independence of quality assurance agencies is a vital requirement in the European 
Standards and Guidelines. The question of independence is the focus of standard 3.6; 
but also standard 3.2 regarding official status, standard 3.4 regarding resources and 
standard 3.5 regarding mission statements are relevant to consider when assessing the 
independence of an agency. 

Independence is not a clear and one-dimensional concept. This is partly due to the 
fact that higher education is government funded and regulated in the Nordic countries 
as in most other European countries. The Standards and Guidelines themselves identify 
several dimensions. 

First, they emphasise that agencies must be independent from ministries, as 
well as from higher education institutions and other stakeholders. In Denmark, 

Finland, Norway, and Sweden, the agencies are government bodies. The question of 
independence from ministries is thus very relevant. The agencies have quite different 
legal and organisational set-ups, and this indicates that there are several ways to achieve 
the independence and autonomy requested in the Standards and Guidelines. As higher 
education institutions do not have any direct or indirect ownership of the Nordic 
agencies, independence in this respect is primarily a question of autonomy in the 
evaluation processes. 

Second, the Standards and Guidelines make an implicit distinction between legal 
or organisational independence on one hand, and operational independence on the 
other. Agencies are not asked to be independent in the sense that they do not have any 
formal connections with ministries, higher education institutions or other stakeholders. 



The wording contains an underlying understanding that agencies in some way must be 
separated from the political offices of the ministries or the political bodies of the higher 
education institutions. For instance, the operational independence of an agency should 
be guaranteed in official documents. To fulfil this requirement, an agency should at 
least form some kind of organisational entity that can be identified as being different 
to the political offices or bodies. 

But the Standards and Guidelines, first and foremost, emphasise the need for 
independence and autonomy in the operation of the quality assurance processes, and in 
the formulation of conclusions and recommendations in reports. This implies that the 
definition and operation of procedures and methods, the nomination and appointment 
of external experts, and the determination of the final outcomes, e.g. conclusions and 
recommendations, are undertaken autonomously and independently. 

Standard 3.6 and the attached guidelines will be discussed under the following four 
headings: 

• Guaranties in official documents 

• Definition and operation of methods and procedures 

• Nomination and appointment of external experts 

• Determination of outcomes 

8.2 Guaranties in official documents 

The agency’s operational independence from higher education institutions and 
governments is guaranteed in official documents, e.g. instruments of governance or 
legislative acts. 

As described in paragraph 4.3, the four Nordic agencies all have a legal basis. The 
legal bases guarantee to different extents the operational independence of the agencies: 

• The legal documents regulating EVA’s activities emphasize in a number of 
ways the independence of EVA. Both the Danish Evaluation Institute Act and 
the ministerial regulations for EVA state in their first sections the formal 
independence of EVA. Section two of the Act describes the agency’s right to 
conduct evaluations on its own initiative. 

• The legislative basis of EINHEEG contains no explicit provisions indicating the 
independence of the agency. The legislation dates back to the 1990s and no longer 
reflects the developments that have taken place in European quality assurance. It 
is currently under review by the Ministry of Education. The operation under the 
Ministry of Education has guaranteed the agency operational independence. This 
has been an established practice since the beginning. 

• Like other Swedish state agencies, the tasks of HsV are laid down in a specific 
ordinance. The agency is formally independent with regard to methodology and 
decision-making. HsV is also independent in the sense that the agency can 
conduct evaluations and studies on its own initiative. 

• NOKUT’s operational independence is guaranteed by legislation, especially the 
University Act. The Ministry of Education may not instruct NOKUT beyond the 
level authorized by statute or laid down by the ministry in regulations. 

An agency under assessment should provide an account describing to what extent the 
legal basis or other official documents guarantee its operational independence from 


ministries, institutions and other stakeholders. The differences in types of legal bases 
in the four Nordic countries suggest that such an account should contain an evaluation 
of the effectiveness of the existing legal basis in securing the operational independence 
of the agency. Has it proved its effectiveness in protecting the agency sufficiently from 
political pressure and opportunistic behaviour? The assessment should also take into 
account the national traditions. 

Assessors should also examine the clarity of the mandate granted in the legal basis. 

A clear mandate with a clear description of the tasks of the agency can be a valuable 
contribution to operational independence and autonomy. An unclear mandate can open 
a window for ministries, institutions, and other kind of organs with political interests 
to influence operations. 

The mandate is described with different levels of detail in the Nordic countries: 

• Section one of the Danish Evaluation Institute Act states that the main purpose 
of EVA is to contribute to the quality assurance and enhancement of education, 
teaching and learning in Denmark. EVA has the right to initiate quality assurance 
processes for themes, programmes, and institutions, etc., from primary education 
(ground school) to long-cycle higher education programmes (university master 
level). Private schools, doctoral programmes, and educational institutions and 
programmes under the auspices of other ministries are not obliged to participate 
in EVA’s processes. EVA should also advise the ministry, other public authorities 
and educational institutions on questions related to quality assurance and 
development; and EVA also has the task of collecting national and international 
experiences of educational quality assurance and quality enhancement, and to 
renew the applied methods. 

• FINHEEC’s tasks are to provide assistance to higher education institutions and the 
Ministry of Education on issues related to quality assurance; evaluate polytechnic 
accreditation and establishment projects; evaluate activities of institutions 

and higher education policy. FINHEEC should also carry out initiatives 
concerning higher education evaluation activities and their development, engage 
in international co-operation in quality assurance, and promote research in higher 
education institution evaluation. An additional task is to evaluate and accredit 
the professional courses organised by the higher education institutions. Since 
2005, FINHEEC has also been in charge of the evaluation of higher education 
institutions in the autonomous province of Aland. 

• HsV is to contribute to the improvement of quality in the operations of higher 
education institutions. The major responsibilities are: to contribute to the quality 
enhancement of the institution itself (enhancement); to review the education 
vis-a-vis goals and provisions in the Higher Education Act and Ordinance, 
which also involves an accreditation of degree-awarding powers (control); and to 
provide information to students and others to support their choice of educational 
programme or subject (information). The relationship of the National Agency for 
Higher Education to the higher education institutions is laid down in official 
documents. 

• NOKUT was established to supervise and help to enhance the quality of higher 
education in Norway. The purpose of NOKUT is laid down in the Act. The 
body shall be an independent state body which, by means of accreditation and 
evaluation, shall control the quality of Norwegian institutions that provide higher 


education, as well as recognize qualifications awarded by institutions not subject 
to the present Act. Accreditation and evaluation activities shall be designed in 
such a way that the institutions can draw benefit from them in their quality 
assurance and enhancement work. 

A look at the mandates of the Nordic agencies gives a hint as to what constitutes a clear 
mandate. First, the purpose of the agency must be clearly described. This could include 
the balance between control/inspection and more enhancement oriented purposes. 

It could also include a description of the role of the agency in the national quality 
assurance system in higher education, e.g. the definition of NOKUT as the national 
accreditation authority in Norway. It could also include a description of the division of 
labour with other bodies, e.g. ministries, institutions, and other agencies, as well as the 
powers entrusted to the agency, e.g. to oblige institutions to take part in the quality 
assurance processes. 

A clear mandate also implies that the area in which an agency operates is clearly 
described. This could for instance include a description of which types of programmes 
or institutions the agency could make subject to evaluation. It could also include a 
description of the geographical and/or ministerial areas covered by the agency, e.g. 
that FINHEEC is also responsible for quality assurance of higher education in the 
autonomous region of Aland. 

Governments, educational institutions or other formal owners of agencies interact 
with the agencies in a number of ways. If a council or board is part of the 
organisational setup, the owner will typically play an important role in the nomination 
and appointment of council or board members. Therefore, it is important to clarify the 
way this takes place when assessing the operational independence of an agency. 

The diversity of legal foundations of the Nordic agencies also results in different 
kinds of relationships between governments and agencies: 

• EVA is an independent body, outside the government structure, governed by a 
board with responsibility for the overall supervision of the Institute, including 
the annual action plan. The board appoints the executive director who must be 
formally approved by the Minister of Education. The Executive Director manages 
EVA and is responsible to the Board. The Minister of Education nominates and 
appoints the Chairman. The Minister of Education also appoints 9 of the other 
board members upon the recommendation of the ministry’s advisory boards, 
while the tenth member is appointed by the Minister of Science, Technology and 
Innovation. The two ministers formally approve the annual action plan. 

• FINHEEC is governed by a council of 12 persons with established experience in 
evaluation of higher education institutions. The Ministry of Education appoints 
the members for a four-year term, and also appoints one member to chair the 
council, as well as a vice-chair for the term. The ministry can institute special 
subcommittees within the council to prepare issues at hand. The Secretary 
General is head of the secretariat, which is located within the Ministry. The 
council appoints the Secretary General, who is formally approved by the Ministry 
of Education. 

• HSV is led by the University Chancellor who is appointed by the Government. An 
external board, appointed by the Government, makes overall decisions on matters 
of principle. A government bill from 1999 binds HSV to carry out six-year cycles of 


evaluations of all subjects and programmes leading to the award of general degrees 
and professional degrees, including doctoral programmes, but HSV has the right 
to initiate any other forms of evaluation it considers appropriate. The ministry can 
only instruct HSV through government commissions. 

• NOKUT is not part of the government structure and acts independently within a 
given framework of law and ministerial regulation. The ministry may not instruct 
NOKUT beyond what is authorized by statute or laid down by the ministry in 
regulations. NOKUT is led by a board which has the overall responsibility for the 
activity and decisions made by NOKUT. The board has seven members with three 
deputy (substitute) members. The board is appointed by the King (in Council) 
for a period of 2-4 years at a time. The Ministry of Education and Research 
appoints the Board’s chairperson. The Board appoints the director of NOKUT for a 
period of 6 years. The director is responsible for NOKUT’s day-to-day management 
according to the overall directives given by the board. 

These quite different approaches make it obvious that an agency must be able to 
provide a clear description of the relationships and the powers entrusted to the different 
levels of the political and managerial structure. The process of appointing the board or 
council - if any - should have a high degree of transparency. It would be an advantage if 
the relationships between the owners/ministry, the board/council and the management 
of the agency were clearly described in the official documents. 

8.3 Definition and operation of methods and procedures 

The agency undertakes the definition and operation of its evaluation methods 
and procedures autonomously and independently from ministries, higher education 
institutions, and organs of political influence. 

Not all agencies can define their methods and procedures completely autonomously 
or independently. The Nordic agencies have broader or narrower mandates, and their 
tasks are set to a greater or lesser extent, e.g. in their legal bases. This does not 
necessarily affect the degree to which the individual agency should be viewed as being 
independent. For instance, an agency can have the task to accredit higher education 
institutions after predetermined criteria. Another agency can by law be obliged to 
include self-evaluation as a methodological element in all quality assurance processes. 
Although these agencies cannot define and operate their procedures and methods 
completely autonomously, they might be able to do so within the given task and 
mandate. 

An agency can choose to involve a large number of stakeholders in the definition 
of a new cyclic quality assurance scheme, as HsV is currently doing in Sweden, and 
FINHEEC recently did in Finland when deciding on a period of cyclical audits - see text 
box below. An agency can also engage in a broad consultation process when formulating 
new criteria to be applied in the quality assurance processes. This was carried out 
in Norway in 2005, when NOKUT revised its criteria. These kinds of searches for 
consensus should be viewed as a natural element in establishing ownership, even 
though the agency in practical terms does involve ministries, institutions and other 
organs of influence in its definition process. 

Yet another example is where an agency can have obligations to coordinate its quality 
assurance activities with other bodies or authorities. This does not necessarily imply 


restrictions in operational independence if the purpose and procedure is clear. For 
instance, EVA’s annual action plan has to be approved by the ministries, but this is 
primarily a means of mutual orientation in order to avoid evaluations being surpassed 
by planned law reforms. 

An agency under review should not only provide an accurate account of the scope 
of its mandate (see above), but should also present arguments to show that the 
mandate leaves a meaningful room for autonomous decisionmaking when defining 
and implementing procedures and methods. In addition to this, agencies should list 
direct obligations and restrictions concerning its operational independence. The crucial 
question here is whether the agency is actually able to fulfil the role it is supposed to in 
the national quality assurance system, as an autonomous and independent body. 

FINHEEC’s consultation process prior to definition of new audit concept 

In designing the national quality assurance system in higher education, Finland 
has used a wide consultation process involving stakeholders at all stages. FINHEEC’s 
audit model was designed by a task force, with representatives of higher education 
institutions, students, employers and FINHEEC. Stakeholders were also involved in the 
process through numerous seminars where the audit model and manual were discussed. 
This offered a means of collecting feedback, but also helped to familiarise institutions 
with the upcoming audit model. The audit system was piloted at two higher education 
institutions in 2005. Feedback from the pilots was also used to refine the audit model 
prior to its final completion. 

8.4 Nomination and appointment of external experts 

The agency undertakes the nomination and appointment of external experts 
autonomously and independently from ministries, higher education institutions, and 
organs of political influence. 

The capacity to recruit external experts without interference is a fundamental 
element in the overall independence of quality assurance agencies. The guideline 
distinguishes between the nomination process and the final appointment of experts. 

The process of nomination of experts takes quite different forms in the Nordic 
countries. 

• EVA drafts a list of academic and professional profiles to be included in an 
expert team and adds specific names to match the different profiles. The specific 
names are identified through the preliminary study. The Nordic network is often 
involved in recommending Scandinavian experts, and EVA does occasionally ask 
the involved institutions to suggest relevant experts. The management presents 
the list to the board, who will then discuss the names and put them in order 

of priority. When the group has been appointed, the involved institutions can 
express whether they perceive formal conflicts of interest in the composition of 
the group. 

• FINHEEC uses predefined criteria for external evaluation groups. For each 
evaluation, the staff at FINHEEC compiles a list of required expertise for the 
external evaluation group. Following this, suitable experts are located. This 
may include asking for recommendations from other agencies, a search from 
FINHEEC’s own expert database or relying on FINHEEC’s networks. In certain 
evaluation types, the institutions are asked to propose expert names that are 
added to a pool of names, from which suitable experts are chosen, based on the 


predefined criteria. Before the experts are appointed, the institutions subject to 
evaluation are given a possibility to comment on the proposed experts.. 

• HsV specifies the type of assessors that can be considered, and the higher 
education institutions are then allowed to propose members for the panels of 
assessors. HsV supplements the recommendation of the institutions with further 
references and also comes up with its own proposals and makes the final decision 
on appointments. 

• NOKUT’s quality system contains detailed criteria for the selection of experts and 
composition of the expert committees. NOKUT itself finds the experts, and might 
consult, for instance, student organisations in order to get suggestions. Before a 
committee is appointed the institutions will be given the right to comment upon 
the composition of the committee, and a situation where the suggested expert is 
considered to be incompetent may occur. 

None of the agencies involve ministries or other organs of political influence in the 
nomination process, but the extent to which the institutions to be evaluated participate 
in the nomination differs considerably. HsV has experienced that the quality assurance 
activities gain greater legitimacy if institutions have the opportunity to recommend 
names of experts. 

The consultation with institutions, however, only applies to the nomination process. 
The control of the procedures for the appointment of experts lies with the agencies in 
all the Nordic countries: 

• The board of EVA must approve the individual members of each expert team. 

• The Council of FINHEEC formally approves and appoints all experts in evaluation 
groups. 

• HsV always makes the final decision on appointments of experts. 

• The Board of NOKUT has delegated the authority to appoint experts to the 
Director. 

Whereas different stakeholders may be involved in the nomination process, the 
appointment process must be without the influence of third parties, if an agency is to 
comply with the standard. 

8.5 Determination of outcomes 

The agency undertakes the determination of the outcomes of its quality assurance 
processes, e.g. conclusions and recommendations in reports, autonomously and 
independently from ministries, higher education institutions, and organs of political 
influence. 

The guidelines attached to standard 3.6 very clearly state that it is only natural to 
consult the relevant stakeholders in higher education, particularly students /learners, 
in the course of quality assurance processes. Nevertheless, the final outcomes of the 
processes should remain the responsibility of the agency. 

All the Nordic agencies carry out consultation processes before finalising their 
reports: 

• EVA’s consultation procedure implies that institutions may comment only on 
factual errors in the draft report, as well as commenting on the evaluation process 
as such. The consultation procedure rarely leads to changes to conclusions and 


recommendations in the final report. This only happens when the correction of 
factual errors seriously affects the basis for a conclusion or a recommendation. 

• FINHEEC gives institutions subject to evaluation a possibility to correct factual 
errors in the report before publication. 

• HsV circulates the descriptive parts of the draft report to the institutions, which 
are asked to comment on points of fact. Once these comments have been 
submitted, the report is finalised and published. 

• In audits, NOKUT sends the report to the institutions, which are given the 
opportunity to comment on factual errors, before the report is finalized. In 
accreditations, the institutions can comment on the experts’ final report. In 
both audits and accreditations, NOKUT’s Board will produce a final conclusion 
based on the report and the institutions’ comments. The institutions or any other 
stakeholder may not influence the conclusions in the report. 

The consultation processes in the Nordic countries only include the institutions and 
aim primarily at correcting factual errors. None of the agencies invite ministries or 
other organs of political influence to comment on the draft report. 

There are different traditions regarding whether the draft report sent to the 
institutions should include conclusions and recommendations, or whether it should 
only comprise the descriptive parts of the report. When including conclusions and 
recommendations, there is, in principle, a greater risk that the institutions might try 
to influence the final outcomes of the quality assurance processes. Agencies with this 
practice should be prepared to deal with this kind of pressure. 

All the Nordic agencies publish their reports with their respective conclusions and 
recommendations autonomously and independently. The reports are published both on 
the agencies’ websites and as printed versions. 


g. External quality assurance criteria 
and processes used by the agencies 


European standard 3.7: 

The processes, criteria and procedures used by agencies should be pre-defined 
and publicly available. 

These processes will normally be expected to include: 

• a self-assessment or equivalent procedure by the subject of the quality assurance 
process; 

• an external assessment by a group of experts, including, as appropriate, 

(a) student member(s), and site visits as decided by the agency; 

• publication of a report, including any decisions, recommendations or other 
formal outcomes; 

• a follow-up procedure to review actions taken by the subject of the quality 
assurance process in the light of any recommendations contained in the report. 

Guidelines: 

Agencies may develop and use other processes and procedures for particular 
purposes. 

Agencies should pay careful attention to their declared principles at all 
times, and ensure both that their requirements and processes are managed 
professionally and that their conclusions are reached in a consistent manner, 
even though the decisions are formed by groups of different people. 

Agencies that make formal quality assurance decisions or conclusions which 
have formal consequences should have an appeals procedure. The nature and form 
of the appeals procedure should be determined in light of the constitution of the 
agency. 


9.1 About criteria and processes 

The European standard and guidelines concerned with the quality assurance criteria 
and processes used by the agencies contain a number of expectations and requirements. 

First of all, the standard states that the processes, criteria and procedures used 
by agencies should be pre-defined and publicly available. Secondly, the standard 
emphasises a number of elements that the processes will normally be expected to 
include. 

Compared to the formulation of the other European standards, it is interesting to 
note how the formulation of this standard is fundamentally different. The difference 
lies in the wording as well as the level of detail of the requirements expressed in the 
standard. 

The standard lists a number of requirements that should be included in the quality 
assurance processes. In addition, instead of using the wording should include - as in 
the other standards - the wording will normally be expected to include is used. This 
difference in terminology is significant and may leave agencies and potential assessors 



confused about the status of this standard. The choice of formulation gives rise to a 
central question about the extent to which the inclusion of these elements in the quality 
assurance processes should be considered compulsory 

To answer this question it is relevant to refer to the focus of European standards and 
guidelines 2.4 to 2.6, as they stress that the named methodological elements actually 
should be included in external quality assurance processes. 

The standards and guidelines related to criteria and processes have been interpreted 
as covering the following elements, which will be treated in turn in sections 9.2 to 9.7: 

• Pre-defined and publicly available processes, criteria and procedures 

• Inclusion of self-assessment, external assessment by a group of experts, site visits 
and a public report 

• Inclusion of a follow-up procedure 

• Professional management 

• Consistent conclusions and decisions 

• The existence of an appeals procedure 

9.2 Pre-defined and publicly available processes, criteria and procedures 

The processes, criteria and procedures used by agencies should be pre-defined and publicly 
available. 

The emphasis on predefined and publicly available processes, criteria and procedures 
is likely to be seen as a reflection of the general ambition of the European standards, 
namely the ambition to ensure that the professionalism, credibility and integrity of the 
agencies are visible and transparent to stakeholders. 

The quality assurance processes used by the Nordic agencies are predefined and 
publicly available on the websites of the agencies. One example is the audit manual of 
FINHEEC - see text box below. Publication of processes, criteria and procedures on 
the website is likely to be the general rule among other European agencies as well, and 
suggests that an agency’s compliance with this part of the standard can be assessed 
merely by accessing the website of the agency. 

When criteria are used by the Nordic agencies they are also pre-defined and publicly 
available on the websites of the agencies. The extent to which criteria are used does 
however differ among the agencies as well as within each agency depending on the type 
of quality assurance activity in question. A general rule is that (predefined and publicly 
available) criteria - or quality aspects in the case of HsV - are used, as a minimum, 
when a quality assurance process results in formal decisions. 

A central question is whether this selective use of a criteria based approach restricts 
the extent to which the Nordic agencies comply with the standard. To put it differently, 
a central question is whether to interpret the standard as one that demands all quality 
assurance activities to rest on a criteria based approach. This question is implicitly 
answered by European standard 2.3 which specifies that any formal decisions made 
as a result of an external quality assurance activity should be based on explicit, 
published criteria. By doing so, the standard implicitly signals that quality assurance 
activities that do not result in formal decisions need not rest on predefined criteria. This 
conclusion is supported by the statement in relation to standard 3.7 that agencies may 
develop and use other processes and procedures for particular purposes. 


As a consequence, a determination of an agency’s compliance with the standard 
requires the provision of detailed information about the outcomes of the different types 
of quality assurance activities that an agency undertakes, and whether those leading to 
formal decisions rest on (predefined and publicly available) criteria. 


FINHEEC’s Audit manual 

The manual describes the Finnish audit model: its targets, criteria, 
methodologies, and follow-up measures. It explains the entire audit process. 
Thus, it can be used as handbook by FINHEEC, the audit groups and higher 
education institutions preparing for an audit. The manual is publicly available 
on EINHEEC’s website in Einnish, Swedish and English. 


9.3 Inclusion of self-assessment, external assessment by a group of experts, 
site visits and a public report 

The processes will normally be expected to include a self-assessment or equivalent 
procedure carried out by the subject of the quality assurance process; an external 
assessment by a group of experts, including, where appropriate, student member (s), site 
visits as decided by the agency and publication of a report, including any decisions, 
recommendations or other formal outcomes. 

By specifying some specific elements that a quality assurance process is (normally) 
expected to include, the standard provides clear advice on what to look for in the 
assessment of an agency’s compliance with the standard. Moreover, it makes clear that 
assessors should focus on the extent to which each of the processes used by the agency 
being assessed include: 

1. a self-assessment, or equivalent procedure, by the subject of the quality assurance 
process; 

2. an external assessment by a group of experts, including, where appropriate, 
student member(s) and site visits, as decided by the agency; 

3. publication of a report, including any decisions, recommendations or other formal 
outcomes; 

4. a follow-up procedure to review actions taken by the subject of the quality 
assurance process in the light of any recommendation contained in the report. 

Whereas this section will focus on interpretations of the first three elements and what 
to look for in an assessment process, follow-up procedures will be dealt with separately 
in section 9.4. 

The clear specifications of the elements which are expected to be included in any 
quality assurance process undertaken by an agency make it evident what kind of 
documentation one should ask for when assessing an agency’s compliance with the 
standard. In relation to 1), the documentation to request would thus be a description of 
the forms in which the subject of a quality assurance process is involved in the process. 
In relation to 2), the agency’s strategy for the composition of groups of experts, and a 
description of the division of labour between the experts and the agency staff as regards 
the assessments would be required. Einally an account of the extent to which site visits 


form part of the quality assurance processes would be needed. Examples of published 
reports would constitute a relevant source of documentation for an assessment of an 
agency’s compliance with 3). 

Although the specifications of elements to be included in the process aid the 
assessment process, the wording of some parts of the specifications does leave 
substantial room for different interpretations that require a discussion of relevant 
minimum requirements. One example is the formulation “equivalent procedure” in 
relation to self-assessment. This formulation implies that assessors will need to discuss 
what a procedure must contain to qualify for being considered as equivalent to a self- 
assessment. This in turn would require a discussion of the appropriate definition 
of a self-assessment. An analysis of the forms of procedures used by the Nordic 
agencies suggests that a procedure should include, as a minimum, some form of 
reflection on own strengths and weaknesses if it is to qualify for being considered 
equivalent to a self-assessment. With this definition, procedures that only comprise, say, 
a questionnaire comprising “closed” questions - as opposed to procedures including 
qualitative elements - should not be accepted as a form of self-assessment. To assist 
in this assessment, the characteristics of the forms of self-assessment applied to the 
different types of quality assurance processes that the agency carry out, as well as 
examples of self-assessment guides in relation to each form of self-assessment, would 
be relevant to consider. 

The statement that the external group of experts will be expected to include a 
student member “as appropriate” is another example of a part of the standard requiring 
assessors of an agency to agree on a definition or a threshold. At least they will have to 
do so if students are not always to be included in the expert groups. Alternatively, the 
assessors may in such cases decide that the decisive factor should be whether the agency 
being assessed is able to provide convincing arguments for deciding not to include 
students in the expert groups. In any case, a description of the strategy for student 
participation in the groups would be required. An account of any student participation 
in expert groups over recent years could also be relevant. 


HsV’s inclusion of students in expert groups and follow-up on 
their experiences 

HsV attaches central importance to the inclusion of both undergraduate and 
postgraduate students in the panels of assessors. The panels set up by the 
agency between 2001 and 2005 have included both categories of students, and 
a grand total of 209 have participated. During the site visits, the panels meet 
both undergraduate and postgraduate students. Acting as an assessor, or having 
some other involvement in the site visits, offers students one way of exerting 
influence over their programmes. In order to persuade more undergraduate and 
postgraduate students to take part in the evaluations, HsV has improved its 
information to the student unions at the higher education institutions. 

Seminars are held each year with student and doctoral student members of 
expert panels to discuss their experiences. The aim has been to gather opinions 
on how the evaluation process can be developed and improved. HsV’s point 
of departure is that undergraduate and postgraduate student assessors are on 
an equal footing with other panel members. The student assessors have always 
expressed a positive opinion about their task and the process adopted by the 
agency. In general the student members of the panels have reported that they 
enjoy the respect of the other assessors. For some of the panels, the role of the 
chair has circulated, with an undergraduate and postgraduate student sharing 
the post with a professor on the panel. This approach was considered a positive 
experience. 


9.4 Inclusion of a follow-up procedure 

The process will normally be expected to include a follow-up procedure to review actions 
taken by the subject of the quality assurance process in the light of any recommendations 
contained in the report. 

European standard 3.7 emphasises that quality assurance processes are (normally) 
expected to include a follow-up procedure to review action taken by the subject of the 
quality assurance process in light of any recommendations contained in the report. 

By so doing, the standard expresses the expectation that a quality assurance agency 
includes follow-up procedures as part of its quality assurance processes, irrespective of 
whether the agency has been given a formal role as regards follow-up or not. 

The understanding and importance of follow-up is elaborated in the guidelines 
related to standard 2.6. These guidelines include the following statement: 

External quality assurance does not end with the publication of the report and should 
include a structured follow-up procedure to ensure that recommendations are dealt 
with appropriately and any required action plans drawn up and implemented. 

This statement suggests that the focus of a procedure must be considered when 
assessing whether a described procedure qualifies for being considered as a follow-up 
procedure. More specifically the statement suggests that a minimum requirement for 
a follow-up procedure is that it focuses on how the subjects of the quality assurance 
processes deal with the recommendations and fulfil any procedural requirements 
concerning follow up. For this purpose it may be relevant to distinguish between 


procedures that provide information about the effects of quality assurance processes 
and those focusing on the adequacy of quality assurance processes. Whereas the former 
should be viewed as a follow up mechanisms, the latter should rather be seen as quality 
work mechanisms. This is, however, not to say that the two areas of focus cannot be 
integrated in one activity. 

The follow-up procedures of the Nordic agencies comprise the following: 

• Although it is not part of the mandate of the agency, EVA monitors whether 
the programmes and institutions which have been subject to a quality assurance 
activity formulate and publish a follow-up plan, and also asks the institutions if 
they have begun implementing the plan. As regards accreditations of professional 
bachelor education programmes, EVA has a defined role in the follow-up 
procedure in cases where an accreditation results in a recommendation of a 
conditional approval of a programme. In these cases, EVA will be involved in two 
phases. Eirstly, the Ministry of Education will ask EVA to assess the follow-up 
plan sent in by the institution. The institute will evaluate the planned initiatives 
and solutions, and analyse whether implementation of these would be sufficient 
to change the hitherto negative assessments of specific quality criteria to positive 
assessments. On the basis of EVA’s written response, the Ministry will decide 
whether to approve the follow-up plan. Secondly, EVA will evaluate whether 

the institution has implemented the follow-up plan satisfactorily within the time 
frame given by the Ministry of Education. In turn, EVA will submit a new 
report to the Ministry with conclusions as to whether the recommendation of a 
conditional approval can be changed to a recommendation of an unconditional 
approval. EVA plays a similar role in the accreditation of medium cycle higher 
education institutions as university colleges, except that there is no phase one. 

The follow-up procedure moves directly to the evaluation of revised or new 
documentation. 

• In Einland, the higher education institutions themselves are responsible for the 
measures recommended by evaluations, as they are for the quality of their 
activities. An important form of follow-up is the annual result and target 
negotiations between the higher education institutions and the Ministry of 
Education. EINHEEC is not involved in these negotiations. However, a follow-up 
procedure is included as part of EINHEEC’s thematic and programme evaluations 
- see text box below. 

• There is clear and explicit legislation governing HsV’s task of undertaking 
follow-up on completed evaluations, as well as other kinds of follow-up. The 
division of responsibilities is also clear and explicit. HsV’s instructions make 
it clear and explicit that the agency is to undertake follow-up of completed 
evaluations, and that it may implement the evaluations and follow-ups that it 
considers appropriate using what it deems to be appropriate methods. This means 
that it is HsV, the authority that conducts evaluations, that follows up its own 
evaluations. It is also HsV that makes its own independent decisions about the 
form follow-up is to take. Eollow-up is an integral part of the HsV evaluation 
model and can take a number of forms. One is a feedback conference about three 
months after the publication of a subject or program review. Such conferences 
aim, among other things, to discuss views on the implementation of the review 


and discuss the content of the assessments as seen from the perspective of 
representatives from the reviewed subjects or programmes. Another form of 
follow up is one that takes place after one year if reviews have resulted in the 
questioning of entitlement to award degrees. If the weaknesses have not, by then, 
been adequately remedied, the institution will lose its right to award the degree 
in question. Finally, after three years, a follow-up of all reviewed programmes and 
subjects is made. The aim is to acquire information on how the departments have 
dealt with the recommendations made in the report and what improvements have 
been carried out. The follow-up also includes consideration of the overall effects 
of the reviews. 

• The legislation concerning NOKUT does not say anything about follow-up 

procedures. NOKUT has no legal basis or authority to conduct follow-ups and has 
not established such procedures. However, the system of audit and accreditation 
in itself works in such a way that it encourages the institutions to follow up on 
recommendations given in NOKUT’s reports. In the case of audits, this is because 
the institution will have to go through another quality audit if it fails the first one, 
and if the institution wants to retain the right to establish new study programmes. 
In addition, the cyclical nature of the audit regime gives NOKUT a possibility to 
investigate whether the institutions have improved their quality systems in the 
period between the audits. If an application for accreditation is turned down, 
the institution must apply once more to be accredited. The institution will have 
to improve the deficiencies in order to satisfy NOKUT’s criteria. If the experts 
approve to the first application, there is no need for follow-up, because the 
institution is qualified and gets its accreditation. An accredited institution is 
supposed to be of such a quality that there is no need for further investigation. 

Whereas these descriptions show that all Nordic agencies have procedures to review 
actions taken by subjects of quality assurance activities, a comparison of the 
descriptions reveals that the procedures differ substantially among the Nordic agencies. 
The impression is that the differences relate to at least two main factors. 

First of all, the differences in the mandate the agencies have been given as regards 
reviewing actions taken by subjects of quality assurance processes, and thus the division 
of labour between different authorities as regards follow-up, appears to be decisive. 

The impression is that the more an agency bares a formal responsibility for reviewing 
actions taken by the subject of the quality assurance process, the more firm and 
comprehensive are the follow-up procedures applied by an agency. Any assessment of 
the follow-up procedures of an agency will thus have to take into account the mandate 
and formal role of the agency in relation to follow-up. 

Secondly, a look at the practices of the Nordic agencies suggests that the basic 
characteristics of the quality assurance activities affect the extent to which follow-up 
procedures are needed. The nature of the quality assurance processes can automatically 
ensure that the subjects of the quality assurance processes follow-up on the detected 
weaknesses or errors. In NOKUT’s accreditations, the institutions will lose their right 
to take in new students if they do not follow-up on a negative assessment. In this case, 
the follow-up procedures of the agencies are of less importance. The nature of the 
quality assurance activities is, therefore, also important to consider when assessing the 
follow-up procedures of an agency. 


FINHEEC’s follow-up evaluations 

A follow-up procedure is included as part of FINHEEC’s thematic and 
programme evaluations. It is not a second evaluation, but an analysis of the 
impact of the evaluation proper and to what extent the enhancement issues 
and recommendations raised during the evaluation have materialised. 

The follow-ups occur approximately three years after the evaluation report 
has been published. They are always initiated by EINHEEC. A steering group, 
with members representing the most comprehensive expertise in the evaluation 
object or theme, is appointed for each evaluation to plan and conduct the 
follow-up. The follow-up procedure constitutes a survey of the participating 
higher education institutions, a concluding report based on the survey, and 
a closing seminar for the participating institutions. 

Eollow-ups have no formal consequences, but they are deemed to constitute 
an integral part of the evaluation process and improvement of the quality of 
higher education. The higher education institutions have commented in their 
feedbacks that the follow-ups are important tools for the enhancement of their 
operations. 


9.5 Professional management 

The agency should ensure that its requirements and processes are managed professionally. 

The quality assurance activities of an agency are often managed by different individuals 
and groups of individuals, which obviously implies a risk of different levels of 
management professionalism across the activities. This risk is likely to exist particularly 
when the management of quality assurance activities - or parts hereof - are 
subcontracted to individuals outside the agencies, but it also exists even when they are 
management by a smaller number of agency staff. In order to assure the same high 
level of professionalism in the management of the activities of an agency it is, therefore, 
important that agencies have procedures to assure this. The guideline is likely to be 
seen in this perspective and may thus be interpreted as calling for the existence of 
detailed prescriptions of how the different elements included in the quality assurance 
activities of an agency must be handled. The Nordic agencies are all very aware of 
the importance of procedures to ensure consistent and professional management in 
general. Their comprehensive strategies for staff recruitment and policies for staff 
development, presented in section 8.2, clearly illustrate this. It is also illustrated by 
the fact that all the agencies have in place a number of internal quality assurance 
mechanisms, ensuring that the quality assurance processes are managed in a consistent 
and professional manner: 

• EVA has an Evaluation Handbook describing in detail the procedures EVA has 
adopted and implemented to ensure the quality of the processes related to the 
evaluations, see text box below. The methodology unit provides methodological 
guidance to the evaluation officers responsible for EVA’s evaluations in order 
to ensure a sound methodological quality of the evaluations and a correct 
methodological use of different forms of documentation in the evaluation reports, 
i.e. statistical data and information from interviews, self-evaluation reports and 


site visits. The staff of the methodology unit, administrative staff and also staff 
with special expertise in relation to language and communication are involved in 
different stages of the evaluation processes. This form of project organisation is an 
important part of the way in which EVA assures the professional management of 
its quality assurance processes. 

• All FINHEEC’s evaluation processes follow certain procedures and practices. 

They are documented into FINHEEC’s evaluation handbooks. There is also a 
separate manual for audits (see 9. 2.). Due to methodological variation in the 
evaluation forms, some evaluation forms do not contain all of the procedures. 
More experienced staff members tutor younger colleagues to ensure that an 
evaluation fulfils required procedures. In addition, an important tool for ensuring 
consistency is the training of external experts. 

• HsV’s evaluations are to follow the evaluation procedures and routines that have 
been documented (prior determination of quality aspects, instructions for self- 
evaluation, assessor’s manual, “internal guidelines” and other joint documents), 
i.e. quality assurance is to apply to all elements of the evaluation process, 
including the recruitment of experts. The Agency tries to minimise the number 
of differences in assessments through directives on how reviews should be 
implemented, through common quality aspects, through frames of reference 
formulated by the assessors and through the training of assessors. 

• NOKUT’s staff use detailed procedures in order to quality assure the appointment 
of experts, site visits, report writing, etc. A description of the procedures is 
stated in NOKUT’s regulations and can be found on NOKUT’s website. NOKUT 
ensures that the conclusions are reached in a consistent manner through different 
methods. Among those are the criteria the committees use as a basis, the mandate 
given to the committees, information and training of experts, the cooperation 
between the committees and internal discussions in NOKUT. 


EVA’s Evaluation Handbook 

The evaluation handbook describes how each of the different steps and activities 
in an evaluation process must be handled. The evaluation handbook is electronic, 
which makes revisions easy whenever a procedure is considered outdated 
or irrelevant. Furthermore, the electronic handbook makes it easy to access 
relevant standardised documents such as letters and agendas to be used 
in the process as the book contains direct links to these. The evaluation 
handbook specifies at which stages it is obligatory for the evaluation officers to 
consult staff members with special methodological, communication, language or 
administrative expertise. 


9.6 Consistent conclusions and decisions 

The agency should ensure that its conclusions and decisions are reached in a consistent 
manner, even though the decisions are formed by different groups of people. 

This guideline appears to primarily concern agencies conducting quality assurance 
activities where the number of programmes, or institutions included in each activity, 
implies that the number of, for example, site visits exceeds that which a single group of 


experts can reasonably manage in terms of human resources, a given timeframe, etc. In 
such cases, the risk of inconsistency is high, and mechanisms to ensure consistency are 
needed. In an assessment process, an agency should, in such cases, be able to account 
for the mechanisms it applies to ensure consistency in conclusions and decisions. 

EVA, HsV, and NOKUT all have activities where conclusions and decisions are 
reached by different groups of people. The mechanisms these agencies use to ensure 
consistency are fairly similar and mainly comprise the use of predefined criteria and 
the training of experts. Furthermore, and most importantly, agency staff involved in the 
quality assurance activity concerned are explicitly given the responsibility of ensuring 
that conclusions and decisions are reached with reference to similar interpretations of 
the criteria and threshold values, e.g. through close dialogue with the involved experts 
and in the drafting of reports. 

9.7 The existence of an appeals procedure 

If the agency makes formal quality assurance decisions, or conclusions which have formal 
consequences, it should have an appeals procedure. The nature and form of the appeals 
procedure should be determined in the light of the constitution of each agency. 

The Nordic countries do not have a strong tradition of formal appeals procedures. Only 
Norway has set up a formal procedure: 

• Section 13 of the EVA Act states that institutions can register objections to 
conclusions made by EVA with the Minister of Education. Until recently, the 
conclusions of evaluation reports had no formal consequences for the institutions, 
and the possibility of objecting has not been used. In the accreditations of 
professional bachelor programmes, it is the Ministry of Education that is 
responsible for making the final decision on the basis of EVA’s analyses and 
recommendations. The Ministry of Education has not established a special appeals 
procedure for accreditation decisions, but institutions can use the Ministry’s 
general complaints procedures. 

• Only two types of evaluations conducted by FINHEEC include a formal decision, 
against which a higher education institution may wish to appeal. These are the 
accreditation of professional courses and audits of quality assurance systems of 
higher education institutions. The appeals can be made to an administrative court 
in Finland. 

• There is no formal system of appeal against decisions taken by HsV. According to 
Swedish legislation, only decisions affecting individuals can be appealed against. 
The Higher Education Act lays down the power of the National Agency for 
Higher Education to both question and withdraw a higher education institution’s 
entitlement to award degrees. The Higher Education Ordinance also lays down 
that no appeal may be made against a decision by the Agency pursuant to the 
Higher Education Act and Higher Education Ordinance. 

• An institution can appeal against NOKUT’s decisions on accreditations and 
evaluations of quality assurance systems. An institution cannot file a complaint 
about NOKUT’s academic evaluation of the quality assurance systems. Neither 
can an institution file a complaint about NOKUT’s academic evaluation of an 
application for accreditation. 


The diversity among the Nordic countries indicates that the question of appeals is 
rather new. This is probably related to the fact that quality assurance processes 
have primarily had an enhancement perspective until recently. Only in Sweden have 
accreditation-like practices been in existence for a longer period of time as an element 
in the processes. Norway is the first country to introduce a comprehensive accreditation 
system, and it is also here that one finds the most developed appeals procedure - see 
text box. 


NOKUT's Complaints Committee 

An institution can appeal against NOKUT’s decisions on accreditations and 
evaluations of quality assurance systems. Complaints will be dealt with by 
an independent complaints committee. The committee will only deal with 
decisions in relation to the formal administrative rules governing the 
processing of applications. An institution cannot file a complaint about 
NOKUT’s academic evaluation of the quality assurance systems or about 
an application for accreditation. Decisions of the appeal board may not be 
appealed. 

Governmental regulation (to the Act relating to universities and university 
colleges) states that there shall be an appeals board, consisting of six members 
and personal deputies. Two members shall be students. The Chairman and 
Deputy Chairman shall fulfil the statutory qualification requirements for 
judges of the Court of Appeal. The Chairman shall not be an employee or 
member of the board of an institution subject to the Act relating to universities 
and university colleges. The members of NOKUT’s complaints committee are 
appointed by the Ministry of Education and Research. 


lo. Accountability procedures 


European standard 3.8: 

Agencies should have in place procedures for their own accountability. 

Guidelines: 

These procedures are expected to include the following: 

1. A published policy for the assurance of the quality of the agency itself, made 

available on its website; 

2. Documentation which demonstrates that: 

• the agency’s processes and results reflect its mission and goals of quality 
assurance; 

• the agency has in place, and enforces, a no-conflict-of-interest mechanism in 
the work of its external experts; 

• the agency has reliable mechanisms that ensure the quality of any activities and 
material produced by subcontractors, if some or all of the elements in its quality 
assurance procedure are subcontracted to other parties; 

• the agency has in place internal quality assurance procedures which include 
an internal feedback mechanism (i.e. means to collect feedback from its own 
staff and council/board); an internal reflection mechanism (i.e. means to react 
to internal and external recommendations for improvement); and an external 
feedback mechanism (i.e. means to collect feedback from experts and reviewed 
institutions for future development) in order to inform and underpin its own 
development and improvement. 

3- A mandatory cyclical external review of the agency’s activities at least once every 

five years. 


10.1 About accountability procedures 

A fundamental expectation to institutions of higher education is that they are able 
to demonstrate the existence of well functioning internal quality assurance systems. 
This expectation is, for instance, expressed in part i of the European Standards 
and Guidelines report, which contains a number of requirements for internal quality 
assurance within higher education institutions. To have similar expectations to quality 
assurance agencies is logical and is expressed in standard 3.8, which states that agencies 
should have procedures in place for their own accountability. 

The guidelines related to the standard specify a number of elements that are 
expected to be part of the accountability procedures of agencies. Similar to the 
guidelines related to standard 3.7, these guidelines are also very detailed. The guidelines 
will be discussed in sections 10.2 to 10.8. 

A general observation is that the guidelines attached to standard 3.8 stress some 
specific processes that are expected to be quality assured by an agency, while other 
important processes are not dealt with. Whereas the guidelines, for instance, emphasise 
quality assurance of the work of subcontractors, quality assurance of important 
documents such as the guidelines for self-evaluation, and central processes such as 



site visits and the use of documentation are not mentioned at all. When examining 
compliance with the standard, one should be aware that the guidelines do not mention 
all the processes of an agency that are likely to be the focus of its internal quality 
assurance system. 

A challenge regarding the content of the guidelines is that there seems to be some 
interrelations between the guidelines related to accountability procedures and some 
other standards and guidelines for external quality assurance agencies. An example 
illustrating this will be highlighted in section 10.4. 

Finally, some of the terms used in the guidelines leave substantial room for different 
interpretations. 

This will be elaborated in the relevant subsequent sections, where each of the 
guidelines to standard 3.8 will be treated in turn under the following headings: 

• Published policy for internal quality assurance 

• Evaluation of fulfilment of mission and goals 

• No-conflict-of-interest of external experts 

• Quality assurance of the work of subcontractors 

• Feedback mechanisms 

• Internal reflection mechanism 

• External review. 

10.2 Published policy for quality assurance 

The agency has a published policy for the assurance of the quality of the agency itself, made 
available on its website. 

This requirement emphasises the importance of a systematic and transparent approach 
to quality assurance. A central question is what a document must contain in order to 
be accepted as a policy. The different types of documents that the Nordic agencies refer 
to in their reflection on the extent to which they have a policy for the assurance of the 
quality of the agencies themselves, clearly illustrate that “policy” may be interpreted 
in various ways: 

• EVA includes an extensive focus on internal quality assurance and quality 
enhancement mechanisms in its different strategy documents, where the visions 
and values underlying the quality work at EVA are also presented. The agency 
plans to develop a quality assurance handbook which will, among other things, 
include a presentation of the purpose of EVA’s quality work and the activities 
involved. EVA’s overall strategy is publicly available on its website, and is also 
published and distributed to a wide range of stakeholders. 

• FINHEEC’s Action Plan and strategy provide the general foundation for its quality 
assurance. The agency has established procedures that assure quality, but these 
have not yet been formulated into a separate written quality policy. 

• HsV has a policy for quality assurance of its evaluation activities. This policy has 
been adopted by the University Chancellor and describes how the agency works 
with the internal quality assurance and monitoring of its operations, and with the 
external appraisal of its activities. For more detailed information - see text box 
below. The policy for quality assurance is published on HsV’s website. 

• NOKUT’s quality system builds on, and refers to, the strategy plan of NOKUT. The 
plan states that the quality system shall work in such a way that it unveils critical 


points in the processes and leads to improvement and development. The quality 
system states both the main objectives of, and criteria for good quality work. 
NOKUT’s strategy plan is openly available on the agency’s website. The document 
that describes the quality assurance system is not. 


HsV’s policy for quality assurance 

HsV has a quality assurance policy for its evaluation activities. This comprises 
both internal and external quality assurance. The quality assurance is based on 
practising what we teach, i.e. subjecting the Swedish National Agency’s evaluation 
activities to the same kind of appraisal as the higher education institutions 
undergo, and complying with the same demands as those it makes of the higher 
education institutions. Awareness of the significance of quality assurance for its 
own operations and for the credibility of the agency’s evaluation activities is 
a strength. The policy is published on the agency’s website in a Swedish and 
an English version. This also applies to other information on the department’s 
activities. 

Examples of important parts of the policy are: 

• Evaluations are to follow the evaluation procedures and routines that have 
been documented (prior determination of quality aspects, instructions for 
self-evaluation, assessor’s manual, “internal guidelines” and the other joint 
documents), i.e. quality assurance is to apply to all elements of the evaluation 
process. 

• In appointing the panel of assessors, the risk of conflicts of interest must be 
avoided. 

• Exchange between different kinds of evaluation organisations, both nationally 
and internationally, must be ensured. 

• An analytical summary of the evaluations undertaken during the year, and 
their effects, is to be drawn up. This analysis is intended for the higher 
education institutions, the Government and the general public. 

Internal monitoring: Evaluation projects are to be monitored continuously, 
while in process, through operational review, at project manager conferences 
and at specific information meetings. 

External monitoring: After evaluation projects have been concluded, feedback 
is to be given by the assessors, project staff, higher education institution 
and students. “Feedback” can take several different forms, such as completed 
questionnaires, seminars, special studies, etc. Evaluation activities are to be 
monitored continuously by the Advisory Board with international experts 
appointed by HsV. 

External review: Evaluation activities are to be reviewed by external assessors 
at least once every five years. 


10.3 Evaluation of fulfilment of mission and goals 

The quality work includes procedures which demonstrate that the agency’s processes and 
results reflect its mission and goals for quality assurance (of higher education). 

It is obvious that a basic prerequisite for the ability to comply with this requirement is 
that an agency has explicit missions and goals for its quality assurance activities, and 
thus that an agency complies with ENQA standard 3.5 (see chapter 7). The standard is 
also related to standard 2.8 which asks quality assurance agencies to produce summary 
reports describing and analysing the general findings of their reviews, evaluations, and 
assessments, etc. 

The Nordic agencies employ a number of different procedures to monitor the 
fulfilment of missions and goals. The procedures for each agency comprise the 
following: 

• The annual performance contracts between EVA and the Ministry of Education 
include goals for achievements in relation to the institute’s evaluations and other 
activities. Each of EVA’s units sets goals for its activities on an annual basis, 
with reference to the general strategy of EVA. The goals are decided upon at the 
beginning of each year, and follow-up on their achievement takes place at the end 
of the year. Surveys among external stakeholders focus specifically on the extent 
to which the agency fulfils its mission and goals. 

• EINHEEC has different procedures to monitor the correspondence of its 
operations with its mission and goals: 

- The council recurrently discusses and redefines its policy. In addition, 

the frequent communication between the council chairs and the Secretary 
General/ Secretary ensures that the staff and the council are aware of each 
others’ activities. 

- Annual negotiations with the Ministry of Education and Rectors’ Councils 
of both higher education sectors provide relevant stakeholder feedback on 
EINHEEC’s operations. EINHEEC has also conducted several large analyses of 
its evaluations. 

- Self-evaluation is used as a reflection tool for determining whether there is 
correspondence between missions, goals and operations. 

- At the end of each term, EINHEEC publishes a report on the past term. It 
contains reflection in retrospect, which is used for planning the next term. 

• HsV evaluates the fulfilment of missions and goals in various ways. Each year, 

HsV reports its fulfilment of missions and goals to the government. Activities are 
internally monitored continually throughout the year. On a regular basis, HsV 
follows up the panels’ views on the evaluations through questionnaires and 
special seminars. Eurthermore, the institutions that have participated in an 
evaluation are invited to a feedback conference to offer their views on the 
outcomes of the process. Special studies are carried out, from time to time, on 
the basis of questionnaires to Vice-Chancellors and officers responsible for quality. 
Eurthermore, a major study has been conducted by an independent researcher on 
the way in which the agency has fulfilled its missions and goals, as seen by the 
institutions and assessors. 


• The Board of NOKUT makes the final judgement in all audits and institutional 
accreditations. Annual self-evaluations are conducted, where elements like feed 
back from institutions and external experts are included. 

The approaches applied by the Nordic agencies illustrate that information about 
the extent to which an agency’s processes and results reflect its mission and goals of 
quality assurance may be obtained by very different means. It can be obtained through 
the collection of the views of external stakeholders, such as the reviewed institutions, 
programmes or policymakers, etc. The collection of views may be obtained through 
both qualitative methods, such as seminars and self-evaluation, and quantitative 
methods, such as questionnaires. Evidence might also be provided through external 
evaluations or summary reports - see text box below. 

Moreover, the form of the procedures is not decisive. What is important is that 
the accountability procedures actually provide information on the extent to which an 
agency’s processes and results reflect its mission and goals. This in turn suggests that an 
agency should be able to point out how the different questionnaires, seminars or other 
activities contribute with information about the fulfilment of the formulated missions 
and goals. 

As the requirement also stresses that the procedures applied should demonstrate 
that the agency actually fulfils its missions and goals, an agency must also be able to 
provide evidence that this is the case. It goes without saying that the methods used 
must be valid and appropriate for measuring the extent to which an agency fulfils its 
missions and goals. 


HsV’s summary reports - How did things turn out? 

The policy for quality assurance states that an analytical summary of the 
evaluations should be undertaken during the year, and their effects are to be 
drawn up. This analysis is intended for the higher education institution, the 
Government and the general public. The report includes an account of the state 
of higher education in Sweden, and is also an important document which shows 
the extent to which HsV fulfils its missions and goals. This report, “How did 
things turn out?” has become one of the agency’s most important and popular 
reports. The report shows the results from the evaluations conducted during the 
year. It also includes sections on: the results and effects of the evaluations; the 
quality of Swedish higher education compared to other countries; the employers’ 
or professionals’ views on the programmes that have been evaluated; and an 
analysis of the composition of the expert panels. The report also suggests what 
the higher education institutions and the government should do to improve 
Swedish higher education. The report is published in Swedish and English. 


10.4 No-conflict-of-interest of experts 

The internal quality work contains procedures which demonstrate that the agency has in 
place, and enforces, a no-conflict-of-interest mechanism in the work of its external experts. 


The expectation expressed in this guideline should be seen in conjunction with the 
guideline related to European standard 3.6, requiring that an agency appoints external 
experts without interference from third parties. The two guidelines are interrelated in 
the sense that an agency’s appointment of external experts without interference from 
third parties appears to be prerequisite for an agency’s ability to ensure that the work 
of the experts is characterised as having no conflicts of interest. However, this is not 
to say that an agency’s full control of the appointment of experts is a sufficient means 
to ensure no conflict of interest exists in the work of the external experts. This control 
will have to be supplemented by some forms of mechanisms to prevent conflicts of 
interests arising, or at least to ensure that immediate action is taken if conflicts of 
interests become apparent. 

All the Nordic agencies apply mechanisms to prevent conflicts of interest arising in 
the work of the external experts. Predefined procedures and criteria for the selection 
of experts and composition of groups of experts are mechanisms employed by all the 
agencies in this respect. Selection criteria include, among others, a requirement that 
the experts do not have any form of relation to the institution(s) or programme (s) under 
review. To ensure that the experts are aware that their independence is a fundamental 
requirement, EVA and NOKUT also demand that the appointed experts confirm in 
writing that they are not in any way associated with the institution(s) or programme(s) 
under review. If they fail to do so, their appointment is redrawn, and another expert 
is appointed. In the case of HsV, potential experts are asked about possible conflicts of 
interest, and references are taken. If such conflicts are found to exist the person is not 
appointed. Once groups of experts have been established, the mechanisms employed by 
the agencies to prevent any conflicts of interests arising vary. As an example, HsV again 
raises the question of conflicts of interest in the 1-2 day training sessions, and the topic 
is included in the handbook for assessors. 

When scrutinising the mechanisms applied by an agency to prevent and avoid 
conflicts of interest in the work of the experts, focus should rest on the extent to which 
the mechanisms are well documented and employed systematically. 


NOKUT’s no-conflict-of-interest mechanisms 

The purpose of NOKUT’s no-conflict-of-interest mechanisms is to prevent 
situations where doubt is cast on a committee’s conclusions because an expert 
is suspected of not being impartial. 

NOKUT’s quality system contains procedures for how to select and recruit 
experts in order to prevent conflicts of interest. Among these are the evaluation 
subject’s right to comment on an expert. In addition, NOKUT has detailed 
criteria on the selection of experts and the composition of groups. All experts 
also need to confirm - in writing - that they do not have any appointments with 
the institution or programme that is the subject of the evaluation, or any other 
connection that may cast doubt upon their impartiality. 


10.5 Quality assurance of subcontractors 

The internal quality work includes procedures which demonstrate that the agency 
has reliable mechanisms to ensure the quality of activities and material produced by 
subcontractors. 

All the Nordic agencies use subcontractors for travel planning, catering services, the 
printing of reports or other practical purposes. These kinds of subcontractors should 
not be in focus when comparing the practices of an agency to the requirement. 

The guideline must be aimed at activities and materials that have a direct link 
to the central elements of the quality assurance processes, e.g. the gathering of 
information and documentation, site visits, assessment, formulation of conclusions and 
recommendations, report writing, and follow-up procedures. 

Among the Nordic agencies, only EVA uses subcontractors in its quality assurance 
processes on an every day basis. The subcontractors used by EVA are consultancy firms 
who carry out user surveys, which often form part of the documentation in EVA’s 
evaluations. The agency has established an internal unit to assure the quality of work of 
the consultancy firms - see text box below. 


EVA's methodology unit 

The methodology unit plays a vital role in the quality assurance of EVA’s 
production processes. Originally, the unit was established with the main 
purpose of assuring the quality of the co-operation with, and the material 
produced by, consultancy firms that carry out the user surveys which are 
often part of the documentation in EVA’s evaluations. This task is still an 
important one, and it comprises a number of different activities, ranging 
from contract management to the assessment of the quality of the reports. 

In relation to the evaluation process, the unit is today also responsible 
for providing methodological guidance to the evaluation officers responsible 
for EVA’s evaluations in order to ensure sound methodological quality of 
the evaluations and correct methodological 

use of different forms of documentation in the evaluation reports, i.e. statistical 
data and information from interviews, self-evaluation reports and site visits. 

The unit is currently working on developing a methodological handbook, 
and also initiates various kinds of methodological development projects with 
the purpose of ensuring the sound methodological quality of EVA’s evaluations 
and other projects. 


10.6 Feedback mechanisms 

The internal quality work includes procedures which demonstrate that the agency has 
in place both internal (staff, council/board) and external (reviewed institutions, experts) 
mechanisms for feedback on its activities, in order to inform and underpin its own 
development and improvement. 

This requirement indicates that at least three elements must be considered when 
assessing an agency’s feedback mechanisms. Eirst of all, the extent to which feedback 


opportunities are given to relevant internal and external stakeholders. Secondly, 
assessors should examine the focus of the feedback. Finally, it seems relevant to 
examine whether the content of the feedback has been documented in such a way that 
it is possible to use it for development and improvement. 

It is difficult to make a universal definition of the term “stakeholders”, as 
stakeholders have different relations to quality assurance agencies, depending on the 
organisational structure of the agency, the national context in which it operates, etc. 
Similarly, “mechanisms” may take a number of different forms. As the subsequent 
descriptions illustrate, feedback mechanisms are highly prioritised by the Nordic 
agencies, and a number of stakeholders are provided with feedback opportunities 
concerning the activities of the agencies. 

External feedback mechanisms : 

• EVA conducts a number of external surveys in order to monitor its external 
activities on a regular basis, or whenever the need for such a survey is identified. 
Besides the surveys, feedback from key stakeholders is also ensured through 
meetings with the Ministry of Education, the Ministry of Science, Technology 
and Innovation and meetings with the Danish Rectors’ Conference. It is the 
ambition of EVA to establish quality assurance activities in all relevant areas 

of its work. These activities should provide the basis for the accumulation of 
internal knowledge and, thereby, on the one hand monitor internal compliance 
with established policies and, on the other hand, constitute a basis for decisions 
on the alteration of policies and procedures. EVA sees an interesting perspective in 
creating an inclusive and joint expression of the satisfaction with the operation of 
the institute and, not least, in being able to follow the development of this overall 
expression over time - see text box below. 

• EINHEEC collects feedback from external experts and the higher education 
institutions who have participated in evaluations concerning the processes and 
methodologies of the evaluations. A web-based feedback solution has been tested 
for one evaluation, and, in the future, all evaluations will make use of this 
solution. In addition. The Board of Professional Courses has separate feedback 
collection mechanisms. Informal stakeholder feedback is collected at annual 
meetings with the Rectors ‘ Councils and student unions of both higher education 
sectors. The follow-up evaluations also offer feedback on evaluation methods. 

• HsV sends out questionnaires to all those responsible for quality assurance at the 
higher education institutions, to all vice-chancellors and to members of all the 
expert panels. The responses are analysed and action is taken where it is deemed 
reasonable. HsV has also initiated a study which focuses on the results of the 
provision of higher education as presented in the evaluations. Seminars are 
held each year with student and doctoral student members of expert panels to 
discuss their experiences. In order to obtain further points of view on its quality 
evaluation procedures, the National Agency has arranged an annual conference 
for those responsible for quality assurance at the higher education institutions. 
Seminars on the evaluations have been held with the vice-chancellors of the 
higher education institutions, and arranged, for instance, by the Association 

of Swedish Higher Education (SUHE). An international Advisory Board was 
appointed in 2001, and it has monitored the Agency’s implementation of the 


quality evaluations continuously. This group submitted its final report in the 
spring of 2005. A new group is now being appointed. The board consisted of 
five internationally recognised researchers in the field of evaluation. The Board 
not only bases its discussions on the evaluation reports but also on discussions 
with members of the staff of HsV. The Board has presented its observations and 
recommendations in four reports. 

• NOKUT’s units have procedures for feedback from reviewed institutions and 
experts. The units also have mechanisms for getting feedback from external 
associates in their annual self evaluation. These mechanisms shall have focus 
on development and improvement. The units are still developing these feedback 
mechanisms, and they are not complete yet. 

Internal feedback mechanisms: 

• EVA also monitors its internal activities in a number of internal surveys. They 
are conducted on a regular basis, or whenever the need for such a survey is 
identified. These activities should provide the basis for the accumulation of 
internal knowledge and, thereby, on the one hand monitor internal compliance 
with established policies and, on the other hand, constitute a basis for decisions on 
the alteration and improvement of policies and procedures. 

• At FINHEEC, the council and the secretariat have both performed a self- 
evaluation every four years, at the end of each council’s term. The self-evaluation 
has been performed on the basis of needs at that time, e.g. in 2003, FINHEEC 
used INQAAHE’s guidelines for good practice as the self-evaluation template. The 
self-evaluations have been used in planning the next four-year term. In 2005, 

the secretariat began to employ self-evaluation on an annual basis. In addition, 
internal feedback is collected in annual results- and development discussions with 
staff members. 

• At HsV, the project managers of each year’s reviews meet regularly to discuss 
their experiences and problems, and to raise questions of common interest. A 
monitoring meeting between the project group and the Head of Department 
takes place about halfway through each project. After the conclusion of each 
review, there are follow-up meetings with the managers of the department and 
the project group concerned in order to gather experiences from the review and 
the work of the group. A departmental seminar is arranged annually to discuss 
the experiences of the past year’s evaluations. In the light of experiences gained, 
and discussions with the Advisory Board, and on the basis of the outcomes of self- 
evaluations and external evaluation reports, routines are updated continuously. 

• NOKUT’s units also have mechanisms for obtaining feedback from internal 
associates in their annual self evaluation. These mechanisms shall have focus 
on development and improvement. The units are still developing these feedback 
mechanisms, and they are not yet complete. 


The EVA-barometer 

The construction of an EVA-barometer reflects the fact that although the 
external or semi external surveys are concerned with the opinions of different 
stakeholders, many of them include a focus on the same six issues that are of 
prime concern to EVA. These are: the quality of the information respondents 
receive from EVA; cooperation and EVA’s level of service; EVA’s organisation 
of quality assurance processes; the reliability and validity of the chosen 
methods; the outcome of a process; and the final products’ achievement of 
aims, e.g. the quality of the final report. 

Until recently, the results of the many different surveys have been reported 
separately. However, as the range of surveys - and particularly the external 
ones - has become more extensive, EVA has seen an interesting perspective in 
creating an inclusive and joint expression of the satisfaction with the operation 
of the institute and, not least, being able to follow the development of this 
overall expression over time. This is what the so-called EVA-barometer is 
about, and it offers the advantage of making it easier for EVA to gain a current 
overview of relevant areas for improvement. 


10.7 Internal reflection mechanism 

The internal quality work includes procedures which demonstrate that the agency has in 
place an internal reflection mechanism, i.e. the means to react to internal and external 
feedback, and recommendations for improvement. 

This guideline can be viewed as a logical continuation of the part of the guideline 
discussed in the previous section. Whereas the guideline concerning feedback 
mechanisms emphasises that such mechanisms should be designed in such a way 
that they inform and underpin the development and improvement of the agency, the 
guideline cited above, literally, only adds that an agency should follow-up on the 
findings of the feedback mechanisms. Put differently, dissemination of the findings of 
the feedback mechanisms is a fundamental prerequisite for fulfilling this requirement. 
Unfortunately, dissemination of findings is not enough to ensure that the agency reacts 
to the feedback and recommendations for improvement. A more systematic procedure 
for follow-up on findings is needed. 

A description of the practises of each of the Nordic agencies illustrates that such 
systematic follow-up procedures can take different forms and may provide some 
inspiration: 

• The information about the quality of EVA’s work that surveys and other 
quality assurance mechanisms produce is shared by means of EVA’s Intranet 
and regular internal meetings. At these meetings, the results of the surveys 
and the recommendations for improvement they have led to, are presented 
and discussed. In order to ensure correspondence between EVA’s goals and 
improvement initiatives, the use of predefined “satisfaction goals” has proved 
efficient. These goals facilitate the selection of results which need to be the focus 
of formulations of recommendations for improvement, which in turn ensure that 
EVA focuses its improvement initiatives on those areas where improvement is 


most needed. EVA has recently implemented an internal procedure to ensure that 
each recommendation will be followed by a clarification of who is obliged to 
follow-up on the recommendation, by which means and within which deadline. 

• In FINHEEC, both formal and informal feedback is discussed and processed by 
the staff and, when appropriate, the council. Feedback is used for the planning of 
operations, evaluation processes and action plans for the next year/term, as well as 
for the development of evaluation methodology. The self-evaluations performed by 
both the staff and the council also work as reflection tools. 

• At HsV, the outcomes of the internal and external feedback mechanisms, as 
described in 10.6, are continually reflected upon by the staff and the heads of 
departments, and appropriate action is taken after due consideration. 

A major feature of NOKUT’s quality assurance system is reflection. The units are 
supposed to assess: whether the methodologies and procedures are efficient; whether 
the work has been done in accordance with the procedures; and whether the quality 
and results of the work are satisfactory. This also includes reflections on feedback from 
external and internal associates. NOKUT’s annual self-evaluation group conducts an 
assessment of the assessments that have been made in the units’ reports - see text 
box below. 

These different ways of ensuring internal reflection and reaction to feedback and 
recommendations for improvement suggest that an assessment of an agency’s internal 
reflection mechanism compliance should focus on the actual outcome of the reflection 
processes, rather than the content. Moreover, emphasis should be placed on the extent 
to which the agency is able to document that it has followed up on the feedback and 
recommendations for improvement. 


The annual self-evaluation at NOKUT 

NOKUT’s units collect external feedback after each evaluation. This is from 
both the institutions being evaluated and the experts being used in the 
evaluations. The units also hold internal de-briefings after each evaluation. In 
addition, the units collect internal feedback, both during the year and as part 
of the annual self evaluation. Information from these feedback mechanisms is 
used in the annual self assessment conducted by each unit, and this is, in turn, 
used to improve procedures and mechanisms. 

At the NOKUT level, an annual self evaluation group evaluates the way 
NOKUT’s units are conducting their tasks, including how their feedback 
mechanism works, and how the units react to the feedback. In addition, 
internal cooperation between the units is an aspect of the annual self 
evaluation. NOKUT’s annual quality report is presented to the board, and 
the Director is responsible for making a follow-up plan. The units take the 
annual report and the plan into account in their continued quality work. 


10.8 External reviews of agencies 

The agency is subject to external review of the agency’s activities at least once every five 
years. 

The expectation that quality assurance agencies, themselves, undergo an external 
review on a cyclical basis resembles European standard 2.7, requiring agencies to 
undertake external quality assurance of institutions and/or programmes on a cyclical 
basis. The arguments for cyclical reviews are put forward in relation to standard 2.7. 
Here it is stated that: 

“Quality assurance is not a static, but a dynamic process. It should be continuous, and 
not “once in a lifetime”. It does not end with the first review or with the completion of the 
formal follow-up procedure. It has to be periodically renewed. Subsequent external reviews 
should take into account progress that has been made since the previous event.” (Standards 
and Guidelines for Quality Assurance in the European Higher Education Area, p. 22) 

The guideline does not prescribe what the focus of external reviews of agencies 
should comprise, which elements the review process should consist of, or who the 
initiators of a review are expected to be. This may leave assessors of an agency with the 
impression that selection decisions concerning these features are left to the individual 
agencies. As is evident from chapter three of the report on the European Standards and 
Guidelines, a number of expectations do indeed apply in these respects. This chapter 
specifies among other things that external reviews of agencies should be concerned 
with not only the activities of the agency, but also its processes, and that the results 
of the review should be documented in a report that states the extent to which the 
agency complies with the European standards for external quality assurance agencies. 
The chapter also states that the process of the review should comprise a self-evaluation, 
an independent panel of experts and a published report. Follow-up is not mentioned 
explicitly as being expected to be part of the review process, but indirectly it is, as it 
is stated that the responsibility for follow-up on a cyclical review of an agency rests 
first and foremost with the national authorities (i.e. owners of the agency) and the 
agency itself. 

The expectations to external experts reviewing an agency, only state that they should 
be “international experts with appropriate expertise and experience”. In a worst case 
scenario, it may reduce the credibility of reviews if more specific criteria for the 
characteristics of the experts are not agreed upon in a European context. 

Among the Nordic agencies, only EVA and HsV have so far been subject to external 
reviews, and reviews which fulfil the requirements mentioned above. The one the 
Evaluation Department of HsV has been subject to took place in 2003/2004. This 
review was followed up in 2005 and supplemented in a new external review of its 
evaluation activities to establish whether the agency meets the European Standards. 

The report was submitted to HsV in December 2005 and to ENQA in March 2006. 

EVA (and its predecessor EVC) has gone through two large-scale evaluations and one 
smaller one since 1998. The last one was conducted in 2005, and included, as with 
the review of HsV, an assessment of EVA’s compliance with the European standards - 
see text box below. 


External Review of EVA in 2005 

In spring 2005, HsV was entrusted with the task of evaluating EVA. The brief 
was to evaluate the totality of the activities of EVA in three central areas: 
evaluation, knowledge centre activities and revenue-generating activities. The 
strategic considerations of the institute were examined, as well as the processes 
and chosen methodologies. The management and internal organisation of the 
institute were also analysed. The external framework and conditions pertaining to 
the work of the institute were considered in the assessment. It was an explicit aim 
that the evaluation should qualify the agency for inclusion in the planned register 
of quality assurance agencies active in Europe. 

The evaluation was carried out hy a panel of assessors consisting of five 
Nordic experts. HsV acted as the panel’s secretary and was responsible for the 
implementation of the evaluation. EVA prepared a self-evaluation report which 
served as the basis of the assessments of the panel. Supplementary information 
was gathered during a four day site visit to EVA. The staff, management, board 
and committee of representatives were interviewed, as well as chairs of panels of 
assessors, representatives of stakeholders and the Ministry of Education and the 
Ministry of Science, Technology and Innovation. 


